EXPG0000001
EXPG0000001
AlixPartners
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione
14 September 2022
AlixPartners UK LLP
6 New Street Square
London
EC4A 3BF
Executive Summary...
Introduction and background ..
2.1 My biography.
2.2 Background to the Report.
2.3 My instructions
2.4 Scope of work and information relied upon
2.5 Approach ....
2.6 Structure of the Report
2.7 Limitations...
The theory of system design and development.....
3.1 Enterprise Execution.
3.2 Model Components.
3.3 Component Interdependence
3.4 Component Hierarchy
3.5 .16
3.6 17
3.7 18
3.8 Approaches to SDLC
3.9 Possible Points of Failure ...
Introduction to the Horizon IT System
4.1 Overview .
4.2 The Post Office and its branches.
4.3 Services available to customers at Post Office branches
4.4 The Horizon IT System
4.5 Legacy Horizon (LHITS)
4.6 Horizon Online
4.7 Balancing and Roll-over.
ICL Pathway’s error logging and remediation....
5.1 Organisational design
5.2 PinICLs and PEAKs .
5.3 KELs
Materials provided to me
6.1 Overview ....
6.2 PinICLs and PEAKs
6.3 Known Error Logs ("KELS”) ...
6.4 Management Reports (“MRS”) .
Pre-processing of documents .....
7.1 Overview ....
7.2 Analytics workstream
7.3 Document Review workstream
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
10.
11.
12.
13.
14.
15.
16.
17.
18.
Appendix A - Inventory of documents relied on....
Appendix B - Example Attribute Grammar ..
Appendix C1 - Example PinICL..
Appendix C2 - Example PEAK
Appendix C3 - Example KEL..
7.4 Review Approach.
SPM training experienced difficulties during National Rollout ....
Hardware issues were problematic during National Rollout ..
Many Post Office branches were disconnected from the central system during
national rollout.....
Financial concerns were considered by ICL Pathway throughout the time pe:
reviewed ..
The tenuous relationship between ICL Pathway, their sponsors (BA and POCL) and
suppliers were often topics of concern for ICL Pathway's management team...
The persistence of reference data mismanagement degraded the integrity of Horizon
aan a 98
The Horizon IT System Helpdesk was often the root of SLA issues with POCL....... 106
The Horizon SMC was frequently cited for not properly filtering calls to the SSC. This
lack of filtering delayed the SSC from resolving technical problems... . 108
The SSC was overwhelmed with PPs but was earnest in its effort to perform its
duties... eee 112
Acceptance Incidents were a gating issue to the financial success of ICL Pathway. A
persisting issue related to AI 376. woue LB
Payment and receipt imbalances were common symptoms with varied causes..... 131
18.1Background...
18.2Qualitative observations...
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Executive Summary
Lad
1.1.2
1.1.3
1.1.4
11.5
1.1.6
1.1.7
1.1.8
The Horizon IT System envisaged modernising the UK's Post Office branches. This was an
ambitious goal: placing hardware and software in c. 18,000 branch locations to allow
subpostmasters and subpostmistresses ("SPMs") the convenience to reliably store and
transmit electronic records of their daily business activities.
The technical aspects of the Horizon IT System were significant but did not account for all
of the complexities of a successful implementation. There were additional organizational
factors that required attention. Both the technical and organizational dimensions of the
Horizon IT System also required vigilance. An IT system is a “living” entity. It needs care
and attention beyond its initial rollout.
The Horizon IT System had multiple constituencies that needed to be both strategically and
tactically aligned. Sponsors and suppliers all played key roles in defining and delivering the
Horizon IT System.
The Horizon IT System's design and implementation needed to account for real-world
contingencies. Many designs are very elegant, but they only maintain their elegance if the
implementation withstands practical realities and concerns.
The Horizon IT System's user support mechanism needed to assist the users as they
migrated from a paper-based process to a computer-based process or switched from using
one system to another. Continuous training for all versions of the Horizon IT System needed
to be available. The support structure needed to cater to end-users (e.g., the SPMs and
clerks working in the branches) who might struggle to adapt to changes from the manual
processes they had undertaken for many years. The support structure needed to be able to
service a high volume of users. The support structure needed to be designed to adapt to the
needs of the users as the IT system evolved through its different versions.
The Horizon IT System’s internal error resolution mechanism needed to be able to quickly
resolve reported errors through identifying root causes, methodically correcting these errors,
and distributing the remedies in a timely and efficient manner.
The Horizon IT System’s functionality needed to maintain accounting integrity. The Horizon
IT System was the origin of sales and inventory information that flowed into the financial
systems of Post Office Counters Limited ("POCL”). Consequently, it was intended to be a
“source of truth” for these fundamental accounting facts. Any errors deriving from the
Horizon IT System would, if not otherwise rectified, be reflected in all downstream processes
and systems.
Throughout my review, I identified shortcomings in each one of these key areas:
(a) Constituency alignment: The tenuous relationship between ICL Pathway, its
suppliers, and its sponsors were often topics of concern for ICL Pathway’s
management team; the Helpdesk was often the root of Service Level Agreement
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
1.1.9
(b)
(c)
(d)
(e)
("SLA") issues with POCL; Acceptance Incidents (“AI”) were a gating issue to the
financial success of ICL Pathway;
Design and implementation: Hardware issues were prevalent during national rollout;
many post offices were disconnected for extended periods of time; the persistence
of reference data mismanagement degraded the integrity of the Horizon IT System;
User support: SPM training experienced difficulties during national rollout; the
Helpdesk was often the root of SLA issues with POCL;
Error resolution: The System Management Centre ("SMC") was frequently cited for
not properly filtering calls to the System Support Centre ("SSC"); the SSC was
overwhelmed with escalated issues, as captured in PEAKs and PinICLs (“PPs"), but
were earnest in their efforts to perform their duties; and
Accounting integrity: The persistence of reference data mismanagement degraded
the integrity of the Horizon IT System. A persisting issue related to Al 376
(accounting integrity); payment and receipt imbalances were common symptoms
with varied causes.
In my opinion, the stability of the Horizon IT System was affected by these shortcomings.
The sometimes-conflicting expectations between ICL Pathway and POCL introduced a
disruptive element at the management level. The effects of these disruptions manifested
itself throughout the implementation of the Horizon IT System. Other ICL Pathway self-
inflicted wounds included suboptimal support from ICL Pathway’s program for training of
SPMs, the Helpdesk support of SPMs, and the Helpdesk support of ICL Pathway’s error
resolution function. A noticeable symptom of these issues was a recurrent balancing
problem experienced by the SPMs, which directly degraded the accounting integrity of the
Horizon IT System.
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
2.1;
2.2
Introduction and background
My biography
24d
2.1.2
2.1.3
2.1.4
2.1.5
2.1.6
a7
I, Charles Cipione, have been appointed by the Post Office Horizon IT Inquiry (“the
Inquiry”) to act as an Expert Witness to independently review and analyse evidence the
Inquiry has received on the Horizon IT System.
By way of introduction, I am a Managing Director within the Risk Analytics group at
AlixPartners. I have been a Managing Director at AlixPartners for over fifteen years. I have
over thirty years of experience in information technology.
I started my career at Arthur Andersen within their Information Systems Risk Management
business unit where I performed various general controls and application controls reviews.
At Arthur Andersen I also developed and implemented various database applications and
analyses related to litigation and bankruptcy clients.
I left Arthur Andersen to start my own consulting venture, Cipione & Associates. This
venture designed, developed, and maintained commercial software. This software was
originally DOS-based (pre-Microsoft Windows) but was then versioned to migrate to the
Microsoft Windows platform. My preferred development environment for Microsoft Windows
applications was Microsoft Visual Basic.
In 2001 I joined AlixPartners to help establish the Claims Management Services business
unit. Our responsibility was to develop and operate systems to support the reporting
responsibilities of U.S. Chapter 11 (bankrupt) clients. Examples of these clients include
WorldCom and General Motors. This involved interrogating, collecting, and organizing vast
amounts of disparate financial and operational data from our clients’ systems. I am the
original architect of AlixPartners suite of claims management systems. These systems are
currently still utilized.
Utilizing my software design, development, and implementation background, I also have
been retained by clients to provide factual and expert testimony regarding the efficacy of
application systems and the management and analysis of data sets pertinent to various
litigation and regulatory issues.
I hold a Bachelor of Science in Chemistry and a Master of Business Administration from
Texas A&M University.
Background to the Report
2.2.1
2.2.2
My review was conducted between the months of June 2022 and September 2022. I have
been supported by a team from AlixPartners.
The evidence I have reviewed has been provided to me by the Core Participants, principally
Fujitsu Services Limited (“Fujitsu”), in response to formal requests made by the Inquiry
Legal Team. In addition, the Inquiry Legal Team have directed me to certain public
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
2.3
2.2.3
2.2.4
documents that have provided me with background information on the Inquiry as well as
background information, from prior legal judgements, on the Horizon IT System. As part of
my review, I have had the opportunity to pose questions to the Inquiry team, which have,
where appropriate, been passed to the relevant Core Participant to respond to. A list of the
information that I have relied on as part of my review is included in Appendix A.
Based on my review of the available evidence I have produced this expert report, which
contains my observations and conclusion ("the/my Report") and I will provide oral expert
testimony (“the/my Testimony”) to the Inquiry on this in due course.
The intended audience for the Report is primarily the Chair of the Inquiry, and as part of
this it will be made available to the Core Participants in the Inquiry as well as to the public.
My instructions
2.3.1
2.3.2
23.3
2.3.4
My work has been informed by a formal set of instructions provided to me, in two parts, by
the Inquiry Legal Team:
(a) An initial set of instructions dated 27 May 2022 which were provided to me on 02
June 2022.
(a) An addendum to these Instructions dated 27 July 2022 which were provided to me
on 27 July 2022.
Collectively these two documents are my instructions from the Inquiry (“the/my
Instructions”). Whilst the Instructions provide the basis for determining the scope of my
work, I have, as an independent expert, been responsible for developing my own approach
to responding to the questions posed in my Instructions.
Broadly, my Instructions state that my Report and Testimony should include:
(a) An introduction to the Horizon IT System and other key terms that will assist the
Inquiry in understanding the substance of my Report, and potentially other future
submissions that are made to the Inquiry. I was instructed that this introduction of
the Horizon IT System should be tailored so as to be understandable to the Inquiry,
the Core Participants to the Inquiry, and to members of the public, who may not have
prior knowledge of the Horizon IT System;
(b) Analysis to identify and illustrate any themes in the problems that were being
experienced by users in the period up to and including the roll out of the Horizon IT
System, including how these problems were resolved or escalated, and the key
individuals who were involved in these processes.
(c) Any overall observations or conclusions, that are within my professional expertise,
as to the themes I identify and the potential reasons for these.
The purpose of this Report is therefore two-fold, which I will set out in two distinct parts:
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
2.4
2.3.5
(a) Part 1: To provide an introduction to systems design and development and to the
Horizon IT System specifically, with the express purpose of providing a suitable
foundation of knowledge for an audience that does not have prior knowledge of the
Horizon IT System such that they can better understand both this Report, and the
subsequent phases of the Inquiry; and
(b) Part 2: To set out, in detail, the information I was provided with, how I reviewed
this information, and the observations and conclusions I have reached.
Whilst I was born in Europe, I have spent all of my professional working career in the United
States of America. I am very conscious that the vast majority of the people reading this
Report will be from the UK, and therefore I have adopted UK English spellings and have,
with the help of my UK-based colleagues, attempted to ensure I use a style that will hopefully
be familiar to a UK audience.
Scope of work and information relied upon
24.1
2.4.2
2.4.3
2.4.4
2.4.5
The Inquiry is approaching the hearings in phases and there are seven phases that have
been defined (a full list of which can be found on the Inquiry’s website under ‘Phases of the
Inquiry’ section:). In addition, the Inquiry has identified a list of 218 issues (the Completed
List of Issues”) which reflect the key themes on which the Inquiry intends to focus its
investigative work. The Completed List of Issues is available on the Inquiry’s website’.
My Instructions specifically relate to Phase 2 of the Inquiry, which deals with “Horizon IT
System: procurement, design, pilot, roll out and modifications.” The Instructions also
identified that issues 1 to 28 from the Completed List of Issues are relevant to Phase 2 of
the Inquiry.
The Inquiry has adopted the definition of the Horizon System which was used by Mr Justice
Fraser in his Judgment (No. 6) “Horizon Issues”, being:
“the Horizon computer system hardware and software, communications
equipment in branch and central data centres where records of
transactions made in branch were processed”.
My Instructions adopt the same definition of the Horizon IT System. It is worth noting that
the terms “Horizon system” and “Horizon IT System" are often used interchangeably:
they refer to the same thing, as defined above. In this Report I will use the term Horizon IT
System as I believe this to be a more fulsome description.
In section 4 of this Report I will explain, in summary terms, what the Horizon IT System is,
what it did, how it was structured, and how the system evolved over time. At a very high
level, the Horizon IT System is an Information Technology (“IT”) system that was installed,
in phases, into every Post Office branch in the United Kingdom (UK). The Horizon IT System
was installed into Post Office branches in the period 1999-2000 and is still in use in branches
https: //www.postofficehorizoninquiry.org.uk/key-documents
https: // www. postofficehorizoninquiry.org.uk/publications/completed-list-issues
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
2.4.6
2.4.7
today. Its function was to help digitise certain activities that took place in Post Office
branches, that is to say moving branches’ processes away from paper-based systems to a
system where all transactions undertaken in a branch, as well as the resulting financial
accounts, were recorded electronically in the Horizon IT System. As with any IT system (old
or modern) the Horizon IT System had been updated frequently to add new functions and
to fix some identified issues (commonly referred to as “software bugs”) in the system. Whilst
there have been numerous updates to the Horizon IT System over the years, it is generally
agreed that there have been three major versions of the Horizon IT System:
(a) The original system, which was introduced into branches from 1999 onwards and
which was active until c. 2010. This is commonly referred to as “Legacy Horizon”.
(b) The first major iteration of the Horizon Online system (referred to as “HNG-X”"),
which was introduced in 2010 and was active until c. 2017.
(c) The second major iteration of the Horizon Online system (referred to as “HNG-A"),
which was introduced in 2017 and is still active and in use in branches today.
Part One of this Report will provide further detail on the Horizon IT System and its various
iterations. The important point to note however is that my work, and therefore my Report
and my Testimony, is focused solely on Legacy Horizon, in accordance with my Instructions
and the temporal scope of the documentation that I received.
My work, and therefore the observations and conclusions in this Report, are based solely on
documentary evidence and data provided to me by the Core Participants, being primarily in
this case, Fujitsu. The information I have been provided with are primarily contemporaneous
documentation and data that were created in the period 07 July 1996 to 31 December 2000
and are therefore between 21 and 26 years old as of the date of this Report. A list of the
documentation I have relied on as part of my review is listed in Appendix A.
2.5 Approach
2.5.1
2.5.2
Three primary categories of documents were provided for me to review. These documents
represent the different lenses available to me as I collected observations and developed
themes about the Horizon IT System.
(a) Monthly Reports (“"MRs") -Various internal ICL Pathway management reports as well
as some joint ICL Pathway and POCL implementation reports.
(b) PEAKs and PinICLs ("PPs") - ICL Pathway’s error logging and remediation tickets.
(c) Known Error Logs (“KELs”) - ICL Pathway’s known errors.
I will describe these documents in greater detail further in my Report, but I wanted to
highlight that these documents are my main source of facts for accumulating observations
and synthesising themes. All other documents cited in this Report were used to gain an
understanding of the concepts and terms resident within the MRs, PPs and KELs.
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
2.6
25.3
In addition to reading, I incorporated a series of computer-assisted technologies to assist in
reviewing these documents. These methods will be described later in the Report, but the
volume of information required me to utilise multiple approaches to efficiently manage the
volume of content provided. A brute force approach to reading over 55,000 error log entries
was not only impractical, but inadvisable.
Structure of the Report
2.6.1
2.7.2
273
The balance of my Report is organised in the following manner:
(a) Part One:
(i) A General Overview of Systems Design, Development, and Implementation
(ii) A Description of the Horizon IT System
(iii) A Description of ICL Pathway’s Error Logging and Remediation Policies and
Procedures
(iv) A Description of the Materials Reviewed
(b) Part Two:
(i) A Description of how I organised the materials for further analysis
(ii) An overview of Analyses Performed
(ili) An itemised discussion of observations and themes
(iv) Supporting Appendices
itations
The primary documents in this review generally spanned the time frame of 1996 through
the end of 2000. Up to a quarter of a century has passed since these documents were
written. Many of these documents were intended for internal consumption by ICL Pathway;
consequently, I must account for the authors’ possible intentions and motivations. These
documents were not written with me as the intended audience; they were written for
audiences with an intimate understanding of the technological, operational, and political
considerations of their era.
As the documents provided only relate to the period 1996 to 2000 my review solely concerns
the roll-out of the Legacy Horizon IT System, not that of subsequent iterations of the system
(e.9., Horizon Online).
As my review progressed, more questions came to mind: I asked the Inquiry to provide
more supporting information or clarifications. As this supporting material arrived, it often
generated more questions. This iterative process could have proceeded indefinitely, but
10
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
2.7.4
2.7.5
2.7.6
2.7.7
2.7.8
time constraints limit the amount of “peeling the onion”. A deeper understanding of the
system is always possible, but practical concerns demand a point of closure.
These documents focus either on high-level managerial issues (as captured in the MRs), or
very detailed discussions of specific perceived errors (as captured in the PPs and KELs). They
only provide a partial view of the actual design, development, and implementation of the
Horizon IT System: they do not cover every facet of the technology's inception through
realisation. These documents overwhelmingly described the problems within the Horizon IT
System implementation; their purpose was to rectify problems, not laud accomplishments.
The volume of documents, particularly the PinICLs and PEAKs was high. My review did not
consist of reading every one of these documents as I believe this would not be pragmatic or
an efficient use of time. I used a targeted approach for my review.
The PinICLs, PEAKs, and KELs are very technical. The esoteric nature of these documents
(see example in Appendix C) means that nuances could have been missed. Conclusions
have been reached based on a review of the available material and my interpretation of
these. Others with different or specific experience may have differing interpretations of
these issues.
I have not been able to quantify the frequency of occurrence of specific issues with the
Horizon IT System, other than based on the available materials. As described later in the
Report, the PPs relate to the activities of the third line of IT support. The first and second
lines of IT support are responsible for filtering out and de-duplicating reported incidents so
only a subset reaches the third line; therefore the PPs only present a partial picture of the
reported issues. I was not provided with the records relating to the first and second line of
IT support, which would presumably contain information on all of the IT incidents being
raised by the SPMs.
Most of these documents represent ICL Pathway’s perspective. Other than a few of the
Monthly Reports that were jointly issued by ICL Pathway and POCL, I do not have any insight
into POCL’s view during this time period.
11
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Part One:
Theory of system design and development and
Introduction to the Horizon IT System
12
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
3.
3.1
B.2
The theory of system design and development
Enterprise Execution
3.1.1 To properly understand software systems, it is important to appreciate how they fit into the
overall execution of the enterprise they support. Software systems are enablers, not
panaceas. In the best situations, software applications can decisively improve the execution
of the enterprise's strategy by streamlining operations. This often includes providing
complete and accurate reporting that informs decision makers in a timely manner. In the
worst situations, mismatched expectations and/or faulty designs and implementations
degrade the execution of the enterprise.
3.1.2 The following five component model is intended to illustrate how software systems fit into
the broader execution of the enterprise:
Model Components
Strategy:
3.2.1 The enterprise’s purpose for existence and ability to persist is guided by its strategy. High
level concepts such as mission and purpose statements are the guidelines that inform all
other aspects of the enterprise. The following text is the UK Post Office’s purpose as of the
time of this writing®:
“We're here, in person, for the people who rely on us
Our Purpose has three equally important but distinct parts.
The first - "We're here” - recognises that Post Office is unique as the only
retailer in each nation and every community across the UK. With over
11,500 branches, we're at once universal and yet fundamentally local. And
this is only possible because our Postmasters are here for our customers
- much as Post Office is here to serve and support our Postmasters.
Without our postmasters, there would be no Post Office.
Next - "in person”. Located in communities across the UK, Post Office
remains a vital part of the British high street even as many retailers
continue to leave it. Despite the recent acceleration of digital services
brought about by the pandemic, the simple reality is there will always be
some things that you can’t do easily online - whether sending a parcel or
having a chat face-to-face. Our Purpose highlights the human connection
- the personal touch ~ we offer as a business.
https: //corporate. postoffice.co.uk/en/purpose-strategy/purpose/our-purpose/
13
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
3.3
3.4
And last, but certainly not least, we're there “for the people who rely on
us”. Although the Post Office is for everyone, we know some parts of
society rely on us more than others. For the people that need us most,
we're proud to provide vital, trusted services that allow them to operate
in cash, pay their bills in person, verify their identity, and more as their
needs change.”
Tactics - Business Operations:
3.2.2. To execute the strategy, it is important to have a mature and well-understood set of policies
and procedures. Designing, developing, and implementing the tactical playbooks that
control the day-to-day business operations across all aspects of the enterprise takes
considerable effort. The balance between aspirational goals and realistic constraints is the
responsibility of those put in charge of making “real-world” decisions that affect how an
enterprise is operated.
Software Systems:
3.2.3 Supporting technologies buttress the business tactics defined by the enterprise's policies
and procedures. A software system's sole purpose is to efficiently reinforce the business
operations.
Data Management (Facts):
3.2.4 Software systems collect data (facts) relevant to the operations of the business. The
management of these facts requires alignment of the software systems to the business
operations and anticipates downstream analytics and reporting.
Analytics and Reporting:
3.2.5 The enterprise consumes the facts, as represented by its data, through analytics and
reporting. This component of the model represents how the enterprise understands the data
collected and managed through a series of manipulations and summarisations of the facts.
Component Interdependence
3.3.1 As implied by their definitions, there is a strong relationship between the components of this.
model. The strategy guides the tactics. The software systems support the tactics and
collects the facts. The data management organizes these facts. The analytics and reporting
interpret the facts. The tactics and strategy components then consume these interpretations
and adjusts as necessary based on these interpretations. In a healthy enterprise, this cycle
is transparent. All components understand and support each other.
Component Hierarchy
3.4.1 It is important to understand that a clear pecking order exists between these components.
Strategy guides tactics. Tactics selects software systems based on their ability to conform
to defined business operation requirements. Data management is governed by the design
14
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
3.4.2
specifications of the software systems. Analytics and reporting rely on the rules employed
by the data management function.
Considering this hierarchical relationship, two concepts should be considered that affect a
healthy long-term relationship between the components: adaptability and complexity.
(a) Adaptability:
(i)
(ii)
The downstream components should respond to the requirements of the
upstream components, not dictate them (e.g., the reporting requirements
should not dictate the business's strategy) otherwise, instability would be
introduced vis-a-vis the “tail wagging the dog”.
An example of an unstable situation would occur if the Analytics and Reporting
component took on the responsibility of collecting ancillary data that was not
represented by the requirements defined by the business operations. In this
example, Analytics and Reporting would be infringing on the responsibilities
that should be the purview of other components. This would be a clear
violation of the proper segregation of duties. The collateral consequences of
this type of situation are inefficiencies of communication, maintenance, and
costs.
(b) Complexity:
(i)
(ii)
ili)
Current efficiency and future flexibility benefit from complexity being localised
as far downstream as possible. Strategy should be high-level and easily
comprehended. Tactics should be well-defined and as simple as possible to
achieve strategic goals. Software systems should strictly conform to the
business's operational needs. Data management should be tightly governed
by a set of data definitions. These definitions should anticipate the need for
reference information for future analytics and reporting updates. Analytics
and reporting should assume the responsibility for as much complexity as
possible. This waterfall approach optimizes the responsiveness capability of
the model.
An example of not adopting this philosophy is represented by the Tactics
component requiring the recording of information by the software system that
could easily be derived through downstream methods. Let's consider a
requirement that wanted to isolate the reporting of sales by groups of postal
codes. The business operations might put forth a requirement to the software
system to not only record the postal code, but to also record whether those
postal codes were offshore isles. Given this requirement, the software system
would need to not only create the ability to record this data item, but also
create the appropriate user input screens for the input of these indicators.
Obviously, the offshore isle indicator is strongly aligned with the postal code,
and therefore is redundant with the postal code. This complexity should not
be the responsibility of either the business operations nor the software
15
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
3.5
systems. Rather, this is clearly the responsibility of the data management
component. A reference table located within the data management
component could be created to store this indicator that could then be used by
analytics and reporting. The expected consequences of having this complexity
too far upstream is the increased cost of implementation along with the
possibility of faulty data due to the fact that users can input errors.
Systems Development
3.5.1
3.5.2
3.5.3
3.5.4
3.5.5
At the outset, I want to clarify the distinction between the terms software and system.
Software is a computer program (application) that directs the operation of a computer's
hardware (e.g., monitors, hard drives, printers, networking equipment). System is how the
software and hardware perform in a holistic manner. Unless specifically discussing a
computer program, I will use the more universal concept of a system.
Systems development can be difficult to comprehend. This is understandable. The terms,
concepts, and methodologies are esoteric, myriad, and evolving. Consequently, I am
dedicating a few paragraphs to expound on a few fundamental topics and how they can be
apprehended.
Basics:
(a) At the most abstract level, everything in a system can be characterized as building
blocks: input, processing, and output. Let's consider a simple electronic calculator.
(
) The input is the keypad where you enter the keystrokes “2+2=".
(ii) The processing is the computer chip that performs the arithmetic calculation.
(iii) The output is the panel that hopefully displays “4”.
One might think of a “System” as being an extremely expansive network of these atomic
elements: input, processing, and output.
Hardware devices:
(a) At a tangible level, systems can be categorized by their hardware devices. My
examples span across time and are intended as a representative, not comprehensive.
(i) Input Devices - Keyboards, mice, touch screens, card readers, Storage
Devices
(ii) Processing Devices - Central processing unit (“CPU”) (the “brain” of the
computer)
(iii) Storage Devices - Hard drives, memory (e.g., RAM), CD-ROMs, magnetic
tapes
16
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
(b)
(iv) Communication Methods and Devices - Modems, ISDNs (Integrated Services
Digital Network), the Internet, Bluetooth
(v) Output Devices - Monitors, Printers, Storage Devices
Note: Storage Devices could be either an input or an output. For example, I might
want to save a spreadsheet to my hard drive today, only to retrieve it for further
work tomorrow. Today the hard drive would be considered an output because it is
the destination of my processing. Tomorrow, it would be considered an input as I
retrieve my spreadsheet for further processing. Then, as I save my changes, the
hard drive would once again revert to its output function. In both instances it is a
Storage Device, but Storage Devices can perform both the input and the output
function. I am using this example for the purpose of illustrating the need for
understanding the contextual nature of the use of terms. Even in this very basic
explanation, we can foretell the bleeding of meanings.
3.6 Software Types
3.6.1
3.6.2
As stated above, software is a computer program that directs the operations of the
computer's hardware. There are many different types of software, and they sometimes
interact directly with the hardware, sometimes interact with other pieces of software, and
sometimes they interact with the users.
(a)
(b)
(c)
(d)
Operating System ("OS") Software — The OS is the low-level software that allows all
other software to interact with the computer's hardware. Examples include
Microsoft's Windows, Linux, and Apple's MacOS.
Database Management System ("DBMS") Software - DBMS software specializes in
managing large amounts of data, usually (but not always) in a structured set of
tables. This type of organization allows other programs and users to query and
analyse the data resident within and across these structures. Examples of DMBS’s
include Microsoft SQL Server and Oracle Database.
Application Software - Application software, as the name implies, is software made
for a special purpose and is the type of software computer users interact with most.
Microsoft's Word and Excel are examples of application software. The Counter
component of the Horizon IT System (described in detail later in this Report) is also
a type of application software.
Application Development Software - This is a type of software used by people and
teams to create application software. It allows teams to efficiently manage the
division of labour throughout the development process and assists with version
control (the ability to identify the exact code related to each application program
release). Examples of application development software include Microsoft Visual
Studio and Android Studio.
Certainly, there are many other types of software, but these four categories allow me to
illustrate how software types interact with each other.
17
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
3.7
3.6.3
SDLC
3.7.1
3.7.2
Let us consider an accounting application. This piece of software would have been developed
using an application development software and would likely be supported by a DBMS to
record and retrieve the accounting transactions entered by its users. Both the accounting
application and the DBMS will rely on the OS to coordinate all interactions between the
individual software and the hardware.
SDLC can stand for either Software Development Life Cycle or Systems Development Life
Cycle. Software Development Life Cycle is narrowly focused on the development of the
software application. Systems Development Life Cycle casts a wider net and includes the
associated hardware being directed by the software. For purposes of this explanation, I will
be referring to Systems Development Life Cycle.
There are a variety of approaches to SDLC. Different teams determine which is appropriate
based on their situation. I will explain SDLC across seven commonly used stages.
(a) Planning - This stage of the life cycle includes determining what is being requested
and putting together a project plan to deliver the requested system. The project plan
estimates the amount of resources (people, time, and cost) required to complete the
remaining stages of the life cycle.
(b) Analysis - This stage is where the design team gathers as much information as
possible about every detail of the requested system. This covers issues such as
functionality, performance, equipment, and cost.
(c) Design - After the planning stage is complete, the technical design of the system is
documented. This is also the stage where it is determined what functionality will be
designed and developed internally, and what functionality exists externally and can
be purchased and incorporated into the overall solution. If an external resource is
determined to be appropriate, an integration portion of the design will be
documented. This also includes hardware selection. All other stages are dependent
on clear communication between the design team, the client, and the development
team. If there are miscommunications, functionality, time, and money will be at risk.
(d) Development - Using the technical design document from the previous stage, the
development team will transform the design into a functioning system.
(e) Testing - This phase is used to ensure that the results of the development phase
align with the expected functionality, performance, and hardware described by the
technical design document. There are usually two levels of testing.
(i) Quality Assurance ("QA") - Carried out by a separate group of professionals
associated with the development team prior to exposing the system to the
user community. QA must approve the efficacy of the system before
promoting it to the next testing level.
18
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
(ii) User Acceptance Testing ("UAT") - A small group of users from the group
requesting the system then performs “real world” testing to make sure that
the system meets their expectations.
If there are issues at either level of testing, the development team is engaged to
remediate the issues, or to discuss any miscommunications that might be the source
of the identified issues. Oftentimes, there are certain benchmarks that define whether
the system can be promoted to the deployment stage. In other words, the system
does not need to be perfect to be deployed, but it does need to be acceptable to the
user community.
(f) I Deployment - Once the system has been promoted from the testing stage, it is then
ready to be deployed to the wider user audience. This can be done all-at-once, or in
stages. This is usually decided in the planning stage of the SDLC. As the system is
deployed, users also receive documentation and training on the use of the system
along with a contact mechanism for the system’s helpdesk.
(g) Maintenance - Issues missed in testing, new desired requirements, and general
operations questions are identified as a wider audience of users interact with the
system. These are usually captured and addressed by two support groups related to
the development team.
(i) Helpdesk - This is the first point of contact for any user having issues with the
system. General operational questions are addressed directly. Requests for
new functionality are logged. Perceived errors in the system are initially
assessed. If the perceived error goes beyond the ability of the helpdesk to
resolve, it is promoted to the error logging and remediation team.
(ii) Error Logging and Remediation - A perceived error promoted from the
helpdesk is further evaluated by this function. If the perceived error is
deemed to be valid, it is sent to the development team for remedial treatment.
Depending on the type of error, the remediation could be quickly addressed,
or require an indefinite amount of time to resolve. A communication back to
the helpdesk occurs so that the reporting user can be alerted to the status of
the remediation. If the perceived error is considered not to exist, advice is
then reverted to the helpdesk. Important to the process is the ability to track
symptoms and causes as the errors are identified and resolved.
3.7.3 As the last stage of the SDLC initiates, it connects back to the planning stage to assess its
efficiency and to determine if new functionality should be pursued vis-a-vis new versions of
the system. If new versions of system are deemed appropriate, the process starts again.
3.8 Approaches to SDLC
3.8.1 Over time, there has been an evolution of how the stages of the SDLC are modelled. The
oldest model, waterfall, required each step to be performed in sequence. Newer models
(e.g., Agile development) allow for subdividing the system and moving pieces to UAT as
19
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
quickly as possible. These approaches are iterative in nature, but benefit from increased
feedback from users to developers.
3.9 Possible Points of Failure
3.9.1
3.9.2
As you might already have surmised, systems development covers a wide range of
considerations: hardware, software, desired functionality, SDLC approach, clear
communications between constituencies, adherence to timelines and budgets, etc.
Any missteps within or across any of these considerations will create a situation where the
system appears to be (or could in fact be) deficient. Constant attention to all details is a
necessary and understood regimen to delivering a stable system.
20
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4.
4.1
Introduction to the Horizon IT System
Overview
4.1.4
4.1.2
4.13
4.14
The Horizon IT System is an information technology ("IT") system that was implemented
by the Post Office and installed into an estimated 18,000 Post Office branches throughout
the United Kingdom ("UK"), between 1999 and 2000. The system is still in use in Post Office
branches today, although it has been upgraded several times during its 20+ year lifetime.
Later in this section I will describe the Horizon IT System in greater detail, but before I do I
would like to provide some background to the Post Office and the services that its branches
supported, as this background will make it easier to understand the role and purpose of the
Horizon IT System.
There have been a great many changes to the Post Office since 1999/2000, both IT-related
and non-IT related, and it is therefore impractical to encompass all of these in this
background section. This background section focuses on the structure and services of Post
Office branches in the period 1999-2000, being the period in which the Horizon IT System
was being introduced into Post Office branches for the first time.
This section draws heavily on five documents that I was provided with:
(a) the ‘Technical Appendix to Judgment (No.6) “Horizon Issues” (“TABJ")*;
(b) the ‘Horizon System User Guide / Balancing with Horizon Guide’ ("HSUG"), version
1.0 dated 28 July 20005;
(c) the ‘Technical Environment Description’ ("TED"), version 4.8 dated 22 October
20028;
(d) The ‘Horizon OPS Reports and Receipts’ ("MRR"), version 8.0 dated 08 August 2000”;
(e) The ‘HNG-X Architecture - Counter Business Application’ ("HXA”), version 5.0 dated
04 August 2017°; and.
(f) The ‘Horizon Online Induction Training’ presentation ("HOIT”), which is not dated
but is believed to have been produced in around August 2009°.
I have endeavoured to summarise these documents to what I consider an appropriate level
of detail for the Inquiry, but this has necessarily required me to omit some of the extensive
technical details of the system (the HSUG runs to some 819 pages, and the TED runs to
https: //www.judiclary.uk/wp-content/uploads/2019/12/bates-v-post-office-appendix-1.pdf
POLO0038868
FUJ00079645
FUJO0119554
FUJ00118200
POL00089726
21
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4.2
some 476 pages). I trust the readers will forgive, and indeed hopefully appreciate, this
simplification.
The Post Office and its branches
4.2.1
4.2.2
4.2.3
4.24
The Post Office is responsible for operating a network of branches throughout the UK through
which it offers postal and other services to the general public. Although the formal company
name and structure of the Post Office have changed several times over the years, it has
remained, in essence, a government-owned company, and remains so today. For the
purposes of the period that is relevant to this Report, the Post Office had two main formal
names:
(a) Between 1986 and 2001: Post Office Counters Limited (“POCL"); and
(b) From 2001 onwards: Post Office Limited ("POL")
In this Report where I refer to a “Post Office” I am referring to the physical branch and the
activities that took place there, rather than the actual organisations described above. Where
I do need to refer to the Post Office organisation, for ease and to avoid confusion, I will refer
to this as POCL.
A Post Office branch is a physical location (albeit mobile branches also support some rural
communities) that members of the general public can visit in order to use various services.
Post Office branches can be classified into three broad groups, depending on who was
responsible for operating them:
(a) Crown Post Office branches: these branches are directly managed by POCL and are
known as “Crown” post offices. They are run by employees of POCL (commonly
known as “Crown Office” employees).
(b) Agency Post Office branches: these branches are owned by SPMs who are agents of
POCL. These were typically located within a shop or other small business. The SPM
receives payment from POCL for running the local branch, with their level of
remuneration depending upon the amount of business which is performed at the
branch. In some cases, the SPM would be supported by a manager or assistant.
(c) Outreach services: these are typically small part-time branches that may use a village
hall or mobile van to provide post office services to communities that might not
otherwise receive them.
The graph below illustrates the overall number of branches and their split between Crown,
Agency, and Outreach for the period 2000 to 2021°°:
https: //researchbriefings.files.parliament.uk/docurnents/SNO2585/SNO2585.pdf
22
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4.3
Figure 4.1 Number of post offices by type 2000-2021
Number of post offices by type
2000 to 2021, Thousands
205
ol
mOutreach Agency mCrown
2000 2002 2004 2006 2008 2010 2012 2014 2016 2018 2020
4.2.5 The figure above shows that that vast majority of Post Office branches are Agency branches.
However, a POCL statement in 2003 indicated that, despite Crown branches only
representing some 3% of branches by number, they accounted for over 20% of the
transaction volume". In 2000 the Post Office had around 28 million customer visits each
week, across its branch network’.
Services available to customers at Post Office branches
4.3.1 A Post Office branch provides a wide range of products and services to its customers. It is
estimated that a Post Office branch offered in excess of 170 different products and services’.
Examples of services that a customer can use at a branch include:
(a)
(b)
(c)
(d)
(e)
(f)
Send parcels
Purchase stamps
Purchase lottery tickets
Pay utility bills (such as British Telecom ("BT") telephone bills)
Pay their car (vehicle) tax
Withdraw and deposit cash into their Girobank** account
://publications.parliament.uk/pa/cm200203/cmselect/cmtrdind/7 18/718we17.htm
assets. publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/31809/10-
1260-securing-the-post-office-network.pdf
https://publications.parliament.uk/pa/cm200203/cmselect/cmtrdind/7 18/718we17.htm
at this point in time Girobank was owned by Alliance and Leister, now part of the Santander Group
23
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4.3.2
4.3.3
4.3.4
(9) Withdraw and deposit cash into their account that they hold at certain high-street
banks.
The above are all examples of transactions that a customer could complete in a Post Office
branch. In this Report, when I refer to “transactions,” I am referring to any event in which
a customer used a Post Office service in a branch that needed to be recorded in a system.
For example, when a customer purchases stamps and pays for this purchase in cash. These
transactions would not include a customer doing something that did not result in the need
for this to be recorded in a system. For example, making an enquiry about the cost of
stamps. These transactions would also specifically not include the customer making a
purchase of items from the shop (such as bread or milk) in which the Post Office branch
happened to be located in.
Customers could pay for their transactions using several different methods, such as:
(a) Cash
(b) Debit card
(c) Cheque
A Post Office branch would commonly have a shop associated with it, typically selling
everyday items, such as bread, milk, and newspapers. Transactions associated with the shop
would be kept separate from that of the Post Office branch, as they were, in effect, run as
two separate businesses. In fact, it was common for branches to have two physically
separate counters, one for shop transactions and one for Post Office transactions. Each
counter would have its own till for recording transactions and keeping cash and receipts. If
a customer wanted to buy a loaf of bread and also pay a BT telephone bill, they would need
to complete two separate transactions, each at different tills.
Figure 4.2 A Post Office branch located within a local shop
4.3.5
The majority of Post Office branches were Agency branches, owned and managed by SPMs.
As such, the cash and stock in the branch were owned by POCL but managed day-to-day by
the SPMs. Both cash and stock could periodically be sent centrally from POCL to the Post
Office branches if they were running low and could also be returned if the branch was holding
too much stock or cash.
24
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4
4.3.6
4.3.7
4.3.8
As can be seen from the list of services above, not all transactions that a Post Office offered
were internal to POCL. POCL was also providing services to other entities, both public sector
(e.9., Driver and Vehicles Licensing Agency ("DVLA"), Department of Work and Pensions
(“DWP”) and private sector (e.g., Camelot, BT and Girobank). POCL referred to these other
entities as “Clients”. As such, some of the money transacted in the Post Office branch would
need to be subsequently sent to or obtained from Clients by POCL (e.g., if cash had been
deposited into a Girobank account by a customer of Girobank then POCL would need to send
this money to Girobank).
It was important to keep a record of all transactions that were occurring in the branch, so
that POCL could work out which Clients it needed to pay money to, or claim money from, as
well as ensuring that its cash and stock could be accounted for.
Prior to the introduction of the Horizon IT System, a Post Office branch would record
transactions in paper-form, and/or enter them into their own electronic point of sale
(“EPOS") system. I understand that prior to the introduction of the Horizon IT System some
branches used the ECCO+ EPOS system’.
The Horizon IT System
44.1
4.4.2
4.4.3
The Horizon IT System was installed (also referred to as being “implemented” into all Post
Office branches across the UK. The system was introduced in stages (also referred to as
“rolled out") between 1999 and 2000. The objective of the Horizon IT System
implementation was to modernise the point-of-sale and managerial accounting functions
across the network of Post Office branches. In modern terms this might be described as the
process of ‘digitising’ the Post Office branch network.
The Horizon IT System is still in use in Post Office branches today, although the system has
been updated on many occasions throughout its 20+ year lifetime. Whilst there are many
different upgrades that have been made to the system, it is generally accepted that there
are three major versions of the system:
(a) Legacy Horizon IT System ("LHITS"):
irst introduced in 1999
(b) Horizon Online HNG-X (“HNG-X”): Introduced in 2010
(c) Horizon Online HNG-A ("HNG-A"): Introduced in 2017
Legacy Horizon, when it was first introduced, was known as the Horizon IT System, the
Horizon system, or simply Horizon. I assume that it became known as Legacy Horizon when
it was superseded by Horizon Online in 2010. The following section focuses on the structure
and workings of LHITS. In a later section I will briefly describe my understanding of the two
versions of Horizon Online.
Explanation of Local P.O. Reconciliation and Administration, page 13 (FUJ00079193)
25
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
45 Legacy Horizon (LHITS)
A brief history of LHITS:
4.5.1
The table below briefly outlines some of the key events in the development of LHITS:
Table 4.1 A brief history of LHITS
Date
May 1996
September
1996
November 1997
March 1998
July 1998
October 1998
May 1999
July 1999
Late 1999
Event
The Department of Social Security*® (“DSS”) and Post Office Counters Ltd ("POCL”)
jointly awarded a contract to ICL Pathway Limited (“ICLPL” or “ICL Pathway”) under
the Private Finance Initiative ("PFI”) to develop an IT system that would:
* replace the existing paper-based method of paying social security benefits; and
* automate the national network of Post Offices.
The project was called “the Pathway Project”, and the IT system it was to develop is
variously referred to as “Pathway Horizon”, “Pathway”, “Horizon IT System”,
“Horizon system” or “Horizon”.
At this time ICL Pathway Limited was a wholly owned subsidiary of International
Computers Limited (“ICL”). Fujitsu Limited (“Fujitsu”) acquired 80% of ICL’s shares in
1990, later purchasing the remainder in 1998. ICL was fully integrated into Fujitsu in
2002 and renamed Fujitsu Services Limited.
A limited pilot stage known as “Initial-Go-Live” was implemented in 10 Post Office
branches in Stroud, Gloucestershire. The initial pilot was an interim system for the
payment of Child Benefit only and did not offer full functionality.
The IT system was extended to over 200 post office branches in the North-East and
South-West of England but still only provided for the payment of Child Benefit. The
deadline for completion of the operational live trial of the IT system was missed by ICLPL.
An interdepartmental working group was established to review the viability of the Pathway
Project and the consequences of cancellation. The working group comprised officials from
the Treasury, Cabinet Office, the Department of Trade and Industry (“DTI”) and the DSS.
The interdepartmental working group reported that the Pathway Project remained feasible
but required successful re-negotiation of the contract with ICLPL.
Attempts to renegotiate the terms of the contract between the DSS, POCL and ICLPL
failed.
The original PFI contract awarded to ICLPL by DSS and POCL was terminated. The DTI
announced a new partnership agreement between POCL and ICLPL.
POCL and ICLPL agreed a fixed payment contract to automate the national network of
Post Offices.
The roll-out of the Horizon IT System commenced in Post Office branches nationwide
(referred to as the “National Rollout”).
as Now the Department of Work and Pensions ("DWP"). Note that ICLPL also referred to this as the Benefits Agency
(°BA’) which was an executive agency of the DSS.
26
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Functionality of LHITS
4.5.2
4.5.3
4.5.4
The Horizon IT System delivered in late 1999 was designed to provide two functional
services:
(a) Electronic Points of Sale ("EPOS"): this function was to record, via electronic (rather
than paper) means:
(i) purchases of Post Office products (such as stamps and stationery) made by
customers of the branch; and
(ii) transactions carried out by customers in the branch for the purchase of
products or use of services provided by the Clients of the Post Office (e.9.,
banks, National Lottery, DWP, DVLA etc).
(b) Management accounting: this function effectively provided the SPMs and POCL with
accounts which summarised the cash and stock positions and payments and receipts
activity within a branch.
The data processed by LHITS is high volume (in 2003 POCL stated that Horizon processed
nearly two billion transactions per year*”) but is computationally relatively simple, that is to
say, it does not need to perform complex calculations on the data in order to fulfil its role.
Nonetheless, due to the number of branches, products, and customers it supports, the
Horizon IT System has been characterized as highly complex, but no more complex than
those used by multi-national banking institutions**. It is estimated the Horizon IT System
had over 3.5 million lines of programming code"? and that its documentation runs to more
than 100,000 documents. Fujitsu had publicly stated that when they were awarded the
contract, in May 1996, Horizon was “...Europe’s largest non-military IT contract”.
It is worth noting however that the Horizon IT System was created specifically for the
purpose of servicing the Post Office branches. It did not have the burden of integrating
existing technologies, except where it chose to do so, which limited the possibility of extra
complexity.
‘ious scale and scope
In my view, the project to deliver the Horizon IT System was ambitious in both its scale and
its scope. It is worth reflecting on the state of technology around 1999, when the LHITS was
rolled-out:
(a) The best-selling mobile phone of 1999 was the Nokia 3210”. This had a monochrome
screen, did not support touch-screen navigation, and did not support internet access
https: //publications. parliament.uk/pa/cm200203/cmselect/cmtrdind/718/718we17.htm
TABI, paragraph 16.
Fujitsu case study: https://www.fujitsu.com/uk/Images/postoffice-customer-experience. pdt
TABI, paragraph 15.
Fujitsu case study: https://www.fujitsu.com/downloads/SVC/fs/casestudies/uk-postoffice2.pdf
https: //en.wikipedia.org/wiki/List_of_best-selling_mobile_phones#1999
27
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
(b)
(c)
(d)
(e)
(f)
through a browser. The first iPhone with a touch screen would not be released until
20077,
In 1998 it was estimated that only 31-33% of UK adults had a personal computer
PC”) at home”.
In 2000 it was estimated that only c. 30% of the UK adults had internet in their
homes?’.
Modern social media had not yet been invented (Facebook was only founded in 2004).
The IT sector had invested heavily in addressing the “Millennium Bug” (also known
as the "Y2K" bug). At the time, some systems in use only stored the last two digits
of a year instead of the full four digits (e.g., 99, instead of 1999). If was feared that
at the turn of the new century this could cause code to malfunction. The original
reason for only using two digits was to save memory, as memory was expensive and
limited in the early days of computing.
The prevailing IT development methodology was the waterfall model in which
development progressed monolithically through a linear process, from design, to
build, to test and then to release. The modern concept of agile development was not
mainstream at that time.
Figure 4.3 The Nokia 3210, the best-selling handset of 1999
4.5.6 Some of the aspects of the LHITS development that I believe drove the complexity of the
system and its implementation were:
(a)
The need to design a system that connected all Post Office branches to a central
server but could also withstand a loss of connectivity, without impairing the ability of
https: //www.apple.com/uk/newsroom/2007/01/09Apple-Reinvents-the-Phone-with-iPhone
https://webarchive.nationalarchives.gov.uk/ukgwa/19991013043222/http://www.dti.gov.uk:80/iwfreview/iwfrevie
wi.html
https: //web.archive.org/web/2009090517 1759/http: //www.ofcom.org.uk/research/cm/empdf/emr04_print/cm_200
4.pdf
28
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
(b)
(c)
(d)
(e)
(fy)
(9)
the SPMs to serve their customers and allow the system to correctly synchronise once
connectivity to the central server was re-established.
The need to integrate a variety of software (e.g., Riposte and Tivoli) and hardware
(e.g., touch-screens, printers, communications equipment, bar code scanners, weigh
scales, PIN pads etc).
The need to accommodate hardware failures, as hardware components in the 1990s
were not as reliable as they are nowadays.
A large and diverse user base in the SPMs and the staff they employed, which would
have included varying levels of comfort using ‘modern’ IT systems. Fujitsu notes itself
that “Training was provided to 63,000 staff members from the age of 16 to 87 with
various skills levels.”*° This would have presented, I believe, a significant training,
roll-out and support challenge.
Between August 1999%” and December 2000, over 14,000 branches had LHITS
installed (see Table 4.2 below).
The physical challenges of installing bulky IT hardware into branches.
The need for the system to be very secure, as it dealt with the transfer of money as
well as holding personal details about Post Office customers”.
4.5.7 All of these challenges made, in my view, the design, build and roll-out of the LHITS very
ambitious.
Table 4.2 LHITS Number of Installed Branches
Month
Aug-99
Sep-99
Oct-99
Nov-99
Dec-99
Jan-00
Feb-00
Mar-00
Apr-00
May-00
Cumulative Cumulative
Installed/Live Counters installed
base (Post
Offices)?
321 819
749 1,819
1,596 3,558
1,859 4,122
N/A N/A
1,966 4,413
3,010 6,658
N/A N/A
N/A N/A
N/A N/A
Fujitsu case study: https://www.fujitsu.com/downloads/SVC/fs/casestudies/uk-postoffice2.pdf
ICL Pathway Bringing Technology to Post Office Counters & Benefit Payments - Monthly Progress Report, August 1999
(FUJ00058185)
ICL Pathway Bringing Technology to Post Office Counters & Benefit Payments - Monthly Progress Report December
2000 (FUJ00058197)
TABI, paragraph 80.
N/A indicates where monthly figures are not available from these reports.
29
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Month Cumulative Cumulative
Installed/Live Counters installed
base (Post
Offices)
Jun-00 8,532 18,841
Jul-00 9,814 22,198
Aug-00 11,181 24,674
Sep-00 N/A N/A
Oct-00 13,686 30,025
Nov-00 14,841 32,727
Dec-00 15,142 33,369
LHITS high-level design
4.5.8 In systems considered modern in the late 1990s, architecture often was enacted among
three layers: user interface, business logic, and data. The business logic layer controls the
flow of data to and from the user interface layers. This division of labour sets boundaries of
responsibilities between development teams, consequently speeding the delivery of the
unified product.
4.5.9 These systems, including LHITS, used data-driven logic as much as possible. Data driven
logic is where the mechanisms for computer algorithms refer to values stored in data
structures, rather than referring to application source code. This allows for changes to be
made to the data in the structures, without the need to alter existing application source
code. When applied properly, this approach bypasses source code related testing and
distribution activities, which often are very time consuming.
4.5.10 An example will illustrate how data-driven logic compares to source code encapsulated logic.
4.5.11 Let us consider the sales price for three products: hammer, screwdriver, and pliers. For the
purposes of this example, the sales price for a hammer is £5. The sales price for a
screwdriver is £7. The sales price for pliers are £6. Let us also assume a customer wanted
to purchase two hammers, three screwdrivers, and one pair of pliers.
4.5.12 A computer application could deal with these prices in its source code. I will use pseudo-
code (a plain language description of what the code is supposed to do) to represent how the
source code might work.
Set the total basket amount to £0
If the product is a hammer, then
Multiply the quantity of hammers by £5 and add this amount to the
total basket amount
If the product is a screwdriver, then
Multiply the quantity of screwdrivers by £7 and add this amount to
the total basket amount
If the product is a pair of pliers, then
Multiply the quantity of pliers by £6 and add this amount to the total
basket amount
30
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
If the product is not a hammer, screwdriver, or pliers
Send a series of alert messages to rectify the issue
4.5.13 This process would perform as intended. However, if the sales price for any of the products
changed, a change to the source code would be required. For simple computer applications,
this would be a trivial task that could be performed in a small amount of time. Conversely,
if the application was complex, changing, testing, and deploying the new version could take
a significant amount of time.
4.5.14 Since price changes can be frequent, a data-driven logic approach is appropriate. In this
approach a data table with the item names and their prices is maintained and made available
to the computer application.
Table 4.3 Product Master Table
Product Price
Hammer £5
Screwdriver £7
Pliers £6
4.5.15 The code for the application could now resemble.
Set the total basket amount to £0
For every product purchased
Look for the product in the Product Master table
If the product is found
Multiply the quantity by the price and
add the amount to the total basket amount
If the product is not found
Send a series of alert messages to rectify the issue
4.5.16 In this approach, price changes are dealt with by maintaining the Product Master table:
changes to the source code are not needed. This does imply that the functioning of the
system is reliant on the integrity of information in the Product Master table. If the table is
updated in a timely manner, and maintains its informational integrity, the system is more
responsive to price changes.
LHITS high-level structure
4.5.17 There are many ways to describe the structure of the LHITS, but for simplicity I have
categorised these into four main components:
(a) Counter and peripherals: These were the system components that were located in
branch, consisting of hardware and software;
(b) Communications network: This was the network connection (functionally akin to the
communication role of an internet connection) which allowed data to be sent between
the branch and the LHITS Campuses (see below). This was commonly an ISDN
31
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
connection (a type of communications connection), although some rural branches
used a satellite link;
(c) _ Messaging system: this was the software and protocols responsible for encapsulating
data that was then communicated between branches and the LHITS Campuses (see
below); and
(d) _ LHITS Campuses: these were locations that centrally collected, stored, and processed
data on transactions from across the Post Office branch network and communicated
with POCL’s systems and those of POCL's Clients. These campuses were located in
Bootle and Wigan.
4.5.18 This diagram shows a simplified representation of how these components worked together.
Figure 4.4 Components of LHITS
ClientB Client c
systems systems
The Legacy Horizon IT System (LHITS) i
(D} LHITS Campus
Host layer
Agent layer
Correspondence layer
Communications network
(ISDN/satellite link)
Messaging system sending
data to and from the branch
] tA)
LHITS Counter e+! Counter
in the branch peripherals
(e.g. monitor
keyboard etc.)
4.5.19 The system was designed to operate with an available network connection (“Online
mode”), but also to allow the Post Office branch to carry on serving customers, even if the
network connection was not available ("Offline mode”). In this Offline mode the Counter
was designed to accumulate transactions and synchronize with the broader system once the
communication connection had been re-established. The Counter kept enough information
to perform most tasks, regardless of connectivity status, but not all transactions could be
completed in Offline mode. Two notable transaction types that required the system to be in
Online mode were:
32
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4.5.20
(a) Network Banking Service ("NBS"): Withdrawals of cash from the customer's high-
street bank accounts required an active network connection as funds availability
verification prior to withdrawal was only possible when in the Online mode.
(b) Debit Card Service ("DCS"): Use of debit cards to pay for transactions also required
an active network connection as transaction authorisation was only possible when in
the Online mode.
I will now explain in more detail components A, C and D. A further explanation of component
B (the communications network) is not, in my view, necessary in order to understand the
LHITS. The communications network was provided by a combination of BT and Energis and
was essentially a purchased service that the LHITS made use of, in much the same way as
a household pays for the use of an internet connection.
33
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Component A - Counter and peripherals
4.5.21
4.5.22
4.5.23
The physical components of the Horizon IT System that were located in branch were:
(a) Counter: A personal computer (“PC”) running the Windows NT operating system.
This was loaded with various pieces of Horizon software, most notably Riposte;
(b) Screen: A 10-inch colour touch screen or 12-inch flat panel screen;
(c) Keyboard: A specialised financial keyboard with magnetic stripe reader and Smart
Card reader;
(d) Bar code reader: A handheld bar code reader that could be used to read pre-printed
barcodes on items such as utility bills;
(e) Weigh scales: Scales for weighing postal items (e.g., parcels);
(f) Tally roll printer: A printer used for producing customer receipts as well as some
summary reports for the SPM;
(g) PIN pad: Number pad used by customers for entering their Personal Identification
Number ("PIN") to verify debit card transactions; and
(h) A4 printer: A printer used by the SPM for non-customer administration tasks (e.g.,
printing summary reports).
I understand that approximately 46% of branches had only one Counter installed, with
approximately 33% having two Counters installed, and the remaining having three or more
Counters. Some branches had twenty Counters installed. Across the approximately 18,000
branches in which the LHITS was installed there were approximately 38,000 Counters. At
any given branch there was only one Counter that was connected to the LHITS Campuses;
this Counter was referred to as the “Gateway Counter” or "Gateway PC”.
In order to use a Counter the SPM or a member of the branch staff (collectively referred to
hereafter as “Clerks” or “Counter Clerks”) would need to log in to the Counter using their
assigned username and password.
Single-Counter branches
4.5.24
Smaller branches, such as those located in rural villages, would typically only have one
Counter in them. The diagram below shows the Counter setup in a single-Counter branch,
along with the other devices connected to it (e.g., keyboard, screen etc, collectively referred
to as "Peripherals"). Also included below are photos of the LHITS Counter and Peripherals
in a branch.
34
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 4.5 Counter setup in a single-counter branch
The LHITS Counter and peripherals I
ina single-Counter branch
Back-office
Tally roll
Tallyroll I garcode I printer
reader I (A4 inkjet
printer
or laser)
Components used
I__ Components used __I by the SPMs/Clerks
by the customer
@
Figure 4.6 The LHITS Counter in a branch
Multi-Counter branches
Larger Post Office branches could have multiple Counters in them. The diagram below shows
4.5.25
the Counter setup in a multi-Counter branch, illustrating the importance of the Gateway PC
in providing a connection to the LHITS Campuses.
35
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 4.7 Counter setup in a multi-counter branch
LHITS Campus
I Communications network
I (ISDN/satellite link)
Multi-counter branch Hub
connecting
the Counters '
Counter 1
(‘Gateway PC’)
Counter 3 Counter 2
Back-office
Shared Counter3 Counter 2 Counter 1
be printer
weighing peripheral peripherals peripherals (Aa Inkjet
scales (keyboard etc.) fl (keyboard etc.) ff (keyboard etc.) ae)
4.5.26 Multi-Counter branches could make use of additional LHITS functionality, such as the ability
the transfer an open session between Counters, that is to say if a Clerk starts serving a
customer on one Counter (Counter 2), but then logs on to another Counter (Counter 3), the
current customer session, containing the purchases they have selected so far, is
automatically transferred to Counter 3 and the Clerk is automatically logged out of
Counter 2.
Software - The Riposte Desktop
4.5.27. The Counters ran on the Windows NT operating system, but a user was prevented from
directly accessing Windows. Instead, when logging in to a Counter they were automatically
directed to a piece of software that had been specifically configured for the Post Office, the
Riposte Desktop. This was largely based on a commercial product named Riposte from the
Escher Group. The Escher Group is a separate company from ICLPL and Fujitsu. The Counter
User Interface ("UI") was designed to be as simple and intuitive as possible and is
specifically tailored for use in a retail environment. The intention is that unless absolutely
necessary, the Clerk should not have to type in any data on the terminal. Many transactions
are initiated automatically by the Clerk swiping a magnetic card or reading a bar code using
the Counter's bar code reader. Interaction with the system is achieved by using the keyboard
or the touch screen.
4.5.28 The screen is split into two parts. The left-hand portion contains a number of menu buttons
which are valid in the context of the transaction (though some may be marked with a "stop
sign" which indicates that they cannot be used in this particular transaction.) The right-hand
side of the screen is a "stack" showing, for example, the purchases made by the customer
so far.
36
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 4.8 Screenshot of LHITS’s UI
ABipeste
Quantity
Mransactions
Girobank
aS
Network Banking
¢ r
Ke +;
fep LT 650 [ilBus der
Finish
Stock Units
4.5.29
Modes
4.5.30
‘An important concept to understand is that of Stock Units ("SUs”). A Stock Unit is a unit,
created on the LHITS system to which cash and stock (e.g., stamps and stationery) are
assigned and to which transactions are associated. The Stock Unit mirrors how stock and
cash were physically managed in the branch (i.e., they could, depending on how the branch
was managed, represent the contents of the till tray). Each branch will have at least one
Stock Unit, and multi-Counter branches may have multiple Stock Units, possibly aligned to
the different Counters. Stock Units are a way of managing cash and stock and these Stock
Units can be allocated by the SPM on a medium-term basis, to individual Counter Clerks.
The Counter Clerk is then responsible for ensuring that the Stock Unit balances at the end
of the week, or whenever the Stock Unit is de-allocated from them. I describe the process
of Stock Unit balancing in section 4.7. Stock Units are assigned identifiers such as “DD” or
“AA”, The SPMs can transfer stock between Stock Units using a function on the Counter.
Stock Units can be individual (i.e., assigned to one Counter Clerk only) or can be shared
between multiple Counter Clerks. In some circumstances the SPM may choose to allocate a
Stock Unit to certain specific stock, such as Lottery scratch cards.
Another important concept to understand is that of modes. The Counter supports the concept
of Modes. Examples of modes are "Serve Customer" ("SC"), "Transfer Stock In” ("TSI"),
“Rem Out Supply Division" ("ROSD"), “Rem In Supply Division" ("RISD") and
“Housekeeping” ("HK"). The mode is selected by the user on the Counter applications as
they see fit. The current mode is indicated at the top left-hand corner of the Desktop screen
37
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
(e.g., in Figure 4.8 the Counter is in the “Serve Customer” mode as indicated in the top left-
hand corner of the screen). A Counter is in only one of a small set of predefined modes. A
mode may have the effect of making certain desktop functions unavailable. For example,
the Logout function is not available while the Counter is in Serve Customer mode; the Clerk
has to settle (complete) the current session before they can log out.
Remming in and remming out
4.5.31
Horizon
4.5.32
The Counter supported the activity of recording the movement of cash and stock from the
branch to POCL or vice versa. Stock and cash might be returned to POCL if the branch was
carrying an excess, or certain stock items had expired (e.g., stamps being taken out of
circulation). The term used for accepting a delivery of cash or stock from POCL by the branch
is called “remming in”; the term used for delivering cash or stock from the branch to POCL
is called “remming out”.
Transaction Data
There are three types of transactional data in the Horizon IT System™: Manual entries, TCs,
and Fujitsu entries:
(a) Manual entries are the data entered by the SPMs through the normal course of
utilising the Counter.
(b) Transaction Corrections ("TCs") are produced by the Post Office to be accepted by a
user at the Post Office branch to correct discrepancies in accounting.
(c) Fujitsu entries are injected into the Horizon IT System directly by Fujitsu and may be
used to balance discrepancies.
Coding of the Counter in LHITS
4.5.33
LHITS was coded in Visual Basic, C, and C++. LHITS also utilized Oracle development tools.
and the Riposte product.
Component C - Messaging system
4.5.34
The Counter uses a messaging infrastructure provided by a system called Riposte (and later
WebRiposte), provided by Escher. Everything that Riposte handles is stored as a message.
Messages are constructed using a format known as Attribute Grammar. This is a self-defining
and nested record format that is technology independent. Data fields (or attributes) are not
positional but are identified by a preceding attribute name. They are a tree-like structure
and could be considered a proto-markup language (akin to the XML format we are more
familiar with today). Attributes can be optional and new attributes can be added over time
without existing applications being affected. Applications use just those attributes they are
interested in and are not ‘aware’ of the rest.
% TAB, paragraph 61. A fourth type, “Transactions Acknowledgement” ("TAs") was introduced in 2012.
38
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4.5.35
4.5.36
4.5.37
4.5.38
4.5.39
Let us continue to use our hammer, screwdriver, and pliers transaction to illustrate what a
tree-like structure might look like. As a reminder, a purchase of two hammers, three
screwdrivers, and one pair of pliers is our example.
The tree-like structure might use the relationships.
Sales transaction
Customer
Items Purchased
Item 1
Item 1 Quantity
Item 1 Price
Item 1 Total Purchase Amount
Item 2...
Item 3...
Total Basket Purchase Amount
In our example, this could be represented as follows.
<Sales Transaction Start>
<Customer Name> “John Doe”
<Items Purchased> 3
<Item 1 Name> “Hammer”
<Item 1 Quantity> 2
<Item 1 Price> £5
<Item 1 Total Purchase Amount> £10
<Item 2 Name> “Screwdriver”
<Item 2 Quantity> 3
<Item 2 Price> £7
<Item 2 Total Purchase Amount> £21
<Item 3 Name> “Pliers”
<Item 3 Quantity> 1
<Item 3 Price> £6
<Item 3 Total Purchase Amount> £6
<Total Basket Purchase Amount> £37
<Sales Transaction End>
An example of an Attribute Grammar is contained in Appendix B.
Normal transactions at the Counter take place within a customer session. Each physical
transaction with the customer (e.g., stamp sale, benefit book encashment, postal order sale)
results in the creation of one or more messages depending on the complexity of the
transaction. For example, a stamp sale has one message, and a postal order results in two
messages (one for the postal order and one for the fee). None of these messages is normally
written to the message store until the customer "settles" the session. This results in an
additional transaction for each Method of Payment ("MoP") used. A key feature of each
session is that they are ‘zero-sum’ that is to say the debits and credits of the transactions
must sum to zero (e.g., if a session has transactions for the purchase of £5 of stamps and
£2 of envelopes then the same session must contain a payment, such as cash, for £7).
39
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4.5.40
4.5.41
4.5.42
4.5.43
When a Clerk completes a session then the resulting transactions, in the form of Attribute
Grammars, are saved locally on the Counter in the “Counter Message Store” (also known
as the “Riposte Message Store”), a local repository of all transactions executed on that
Counter. Where there is more than one Counter in an Outlet, the Riposte Message Store is
replicated across all of the Counters. Where there is only one Counter, the Counter contains
two mirrored disks, one fixed and one exchangeable, so that the message store can be
recreated on a replacement Counter if necessary (e.g., in the case of a hardware failure).
Riposte had a data replication facility. In the event the wide area network was unavailable,
the Counter would accumulate messages in the Counter Message Store until the
communication facility was reconnected. When the reconnection was established, the
Counter’s messages would be synchronized with a version of the message store saved on
the LHITS Campuses, in their Correspondence Servers (the “Correspondence Server
Message Store”).
Much of the data required by Counter applications, including much of the way in which they
operate, is passed in “Reference Data”, which is distributed via the same Riposte
messaging mechanisms. Reference data originates with either:
(a) POCL: reference data from POCL providing, for example, product lookup data
containing prices etc.;
(b) Client: reference data from Clients containing information specific to them, such as
Stop Lists?2; or
(c) Application: reference data from LHITS itself which tells the system how to operate
(e.g., what options to make available in certain screens).
This Reference Data is distributed to all relevant branches. It enables the construction of
"soft centred” (data driven logic) applications whose operation can very easily be modified
by changes to the Reference Data (as discussed earlier). A copy of the Reference Data is
saved locally on each Counter in the Counter Message Store. This local copy enables the
branch to carry on operating and completing customer transactions, even if the connection
with the LHITS Campuses is not available (for instance, if the network communication is
lost).
A list of pension and allowance order books on which payment must not be made: HSUG, page 20
40
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 4.9 Counter and Correspondence Message Store
The message store and reference data
LHITS Campus
Correspondence layer
Correspondence Server Message Store
(stores transactional data in the form of
Attributable Grammar messages)
LHITS Counter in the branch
Counter Message Store/Riposte Message Store
POCL reference data
Message Store
(stores transactional data
in the form of Attributable Grammar
messages)
Client reference data
Application reference data
41
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Component D - LHITS Campuses
4.5.44 The LHITS Campuses are actually a collection of different IT components that supported the
‘back-office’ of the Post Office. They were located at two sites referred to as “Campuses”
located in Bootle and Wigan. Resident within the Campuses were several important
functions/processes:
(a) The Correspondence Layer;
(b) The Agent Layer;
(c) The Host Layer;
(d) Reference Data Management System; and
(e) Data Warehouse.
4.5.45 Two important external services* that the LHITS Campus supported were:
(a) Transaction Information Processing; and
(b) I Management Information Services
Figure 4.10 The LHITS Campus
External POCL
and Client Systems
TIP (POCL) MIS (POCL) External interfaces (POCL and Clients)
LHITS Campus
TPS pare Host layer
Warehouse
Agent layer
‘Correspondence layer
Counter layer (located on the Counter in the branch)
4.5.46 The Correspondence Layer handled communications of the network. The Correspondence
Layer and Counter Layer share the use of the Riposte Message Service ("RMS"), a message
storage and replication mechanism, as previously described, which runs on the
Correspondence Servers and the Counter. This supports a shared, distributed message store
[note that it could be argued that sorne elements of these services were contained within the LHITS Campus, but for
simplicity I describe them as external services.
42
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4.5.47
4.5.48
4.5.49
4.5.50
4.5.51
4.5.52
to ensure that information generated at the Counter is replicated in the Campuses and vice
versa. These Riposte mechanisms interact directly with the Agent Layer. A specialised
Riposte Archiver, running on the Correspondence Servers, is used to ensure that all Riposte
messages are written to tape for audit purposes.
The Agent Layer is responsible for transforming the message-based view that is appropriate
for the Counter application into a file view of the Host Layer. It provides facilities to pass
data in both directions: from the Host layer to the Counter, and vice versa. It also provides
facilities to pass messages directly to third party Clients, and to return the Client response
to the Counter. The Host Layer applies any business rules to the information being received
from or sent to the external client system.
The Reference Data Management System ("RDMS") manages all reference data such as
product information and operational elements. POCL’s Reference Data Management Centre
(°RDMC") supports the loading and release of reference data. The Reference Data
Distribution Service ("RDDS") distributes reference data to all branches and data centres.
RDMS is utilised by many elements of the LHITS. It is an example of LHITS’s data-driven
logic; therefore, its integrity is important for the proper operation of LHITS.
The Data Warehouse is a database, or set of databases, used by POCL for querying and
reporting purposes. The Data warehouse is populated from different sets of information
flowing through the hosts.
The Transaction Processing System ("TPS") harvests transactions from the Counters at all
branches and passes them along to POCL’s Transaction Information Processing (“TIP”)
system via the TIP interface.
The Management Information Services ("MIS") is a component built onto the Data
Warehouse to detect errors (including programmatic errors) in the LHITS through a series
of reports and spreadsheets.
TIP was a system used by POCL to collect transaction records about all transactions that
occur at all branches. Transactions are gathered in the first instance by the TPS (once the
Counter has set the End-of-Day ("EoD") marker) which collects all messages that result
from Counter transactions, stock unit balancing and branch cash accounting, and feeds them
into the TPS database, from where they are fed to TIP. TIP was used to feed POCL’s
accounting system and to reconcile transactions with its Clients. While LHITS is not an end-
to-end accounting system, the data it passes to POCL and its Clients must be sufficient to
enable them to balance their own books and settle accounts between them.
43
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Updates to LHITS
4.5.53
I understand that there were several software updates made to LHITS between its initial
roll-out in 1999-2000 and Horizon Online being introduced in 2010. Paragraph 97 of the
TAB) lists these updates; however, little detail is provided as to the nature and content of
these updates. I will highlight a few of these updates.
(a)
(b)
(c)
(4)
(e)
(f)
Starting in August 2000, the Core Systems Release ("CSR") introduced Automated
Payment Service (‘APS”). APS supports payments by customers to utility companies
(e.g., BT, electricity companies, water companies etc) and other Clients of POCL using
bar-coded bills, magnetic cards or smart cards. In addition, this release introduced
reconciliations between the APS data harvested by the APS Agents and the data
harvested by TPS. I understand that there was a subsequent Core Systems Release
Plus ("CSR+"), however I do not have information as to the nature of this software
update.
In February 2001, an upgrade to LHITS, called Maintenance Release M1, was rolled
out. The main purpose of this upgrade was enhancements of the CSR+ applications.
In June 2001, the S06 Release Day D rectifications were released. This included,
amongst other things, a receipts and payments fix.
In 2002/2003 Network Banking Service ("NBS"), Debit Card Service ("DCS"), and
Data Reconciliation Service ("DRS") were introduced.
(i) NBS provides facilities for customers of selected banks and building societies
(those with which POCL had reached an agreement with) to withdraw money
from and deposit funds into their bank accounts.
(ii) DCS enables customers to pay for goods and services using Debit Cards. A
card may be used for some or all of the value of a customer session. The
transaction is verified by online reference to a merchant acquirer who either
approves or declines it.
(iii) I DRS takes information relating to all NBS and DCS transactions and reconciles
the different data flows. It maintains data about transactions until they are
reconciled (this may take some days) and for 90 days thereafter.
I note that the TED states that NBS and DRS were delivered as part of release BI3
and that DCS was delivered as part of release S30.
In 2004 the Post Office Ltd Finance System ("POLFS"), a SAP accounting system,
was implemented. LHITS TPS delivered data directly to POLFS. The following text
comes from an Operations Manual dated December 2006*: “The introduction of the
Post Office Ltd Finance System (POLFS) in Product and Branch Accounting (P&BA),
Chesterfield means that the finance teams can no longer adjust client and/or branch
2 ‘Operations Manual - Branch Trading: balancing and despatch’, version 7 dated December 2006, section 7 (POLO0086704).
44
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
accounts on site. The adjustments have to be made at your branch and are necessary
when branch transaction data does not align with client or supplier data.”.
(g) Although no date is explicitly cited in the TABJ, at some point in 2004, I believe, a
programme called IMPACT was delivered. This programme included several updates
to LHITS*, including:
(i)
(ii)
ili)
(iv)
(vy)
(vi)
rollovers will subsequently be based on a Trading Period (“TP”) that lasts 4 to
5 weeks instead of the weekly Cash Accounting Period ("CAP") used
previously in the LHITS;
non-value stock declarations were removed and stock balancing no longer
checked that such declarations have taken place;
the new concept of Local Suspense account was introduced for the processing
of variances (or discrepancies as they were formerly known);
Stock Units will subsequently no longer be allowed to carry discrepancies over
and any discrepancies will be moved into Local Suspense when the Stock Unit
rolls over;
Additional checks were carried out in order for the final Stock Unit to roll over:
(1) the last Stock Unit was not allowed to roll over if there were outstanding
Transaction Corrections (TCs); (2) Local suspense must be cleared (settled)
before the final stock unit could roll over to the next Trading Period; and
Changes to the data server were made to reduce the number of times that
the message store was scanned to pick up transactions during balancing. A
Riposte mechanism known as “Notifications” was used to add new transactions
to the existing totals as further transactions were generated during the
balancing process?*,
(h) In around 2010, POLFS and SAPADS were merged to make POLSAP. I understand
that SAPADS was POCL’s stock control system (in conjunction with the Logistics
Feeder System (“LFS”)), responsible for ensuring that branches maintained adequate
and appropriate levels of stock and cash.
Impact Release 3 - Balancing and Trading - Statement Production User Interface, Section 2.1.1 and 4.7
(FU}00085125) and Branch Trading Transition Guide, page 28 (POLO0089708).
TABI, paragraph 241.
45
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4.6
Horizon Online
4.6.1
4.6.2
I understand that the original Horizon Online system (HNG-X) was piloted in late 2009 and
rolled out in the spring of 2010. HNG-X replaced LHITS. I understand that there were several
reasons for the release of HNG-X, including:
(a) Reducing the cost of running Horizon: A Post Office branded document titled
“Introducing Horizon Online’ and dated September 2009 contains the following
passage”: “Why is the Post Office® making changes to Horizon. The Post Office in
recent years has made significant losses. The current Horizon application contributes
to this financial position by being a major cost to the Post Office. Horizon Online will
cost significantly less to run, maintain and change. These savings will improve our
commercial position increasing our opportunities to retain current work and to bring
in new business.”
(b) Taking advantage of improved communication technology reliability: The
improved reliability of network technology meant that it was feasible to have
branches that were “online” the vast majority of the time. This benefited a wider
change in the business as an increased proportion of transactions involved NBS and
DCS, which required an active connection to the Data Centre.
(c) Simplify the design of the User Interface: A Post Office branded document titled
‘Introducing Horizon Online’ and dated September 2009 contains the following
passage”: “Horizon Online has a more logical grouping of products and services as
well as fewer product screens to navigate overall. This means that the buttons for
the majority of products and procedures may be found within three screens.”
(d) Simplify business process: I understand that HNG-X also simplified some of the
processes that an SPM was required to complete as part of the back-office
administration of the branch. A Post Office branded document titled ‘Introducing
Horizon Online’ and dated September 2009 contains the following passage: *...the
maximum number of branch reports that may need to be produced has been reduced
from 85 to 44”.
In describing the differences between LHITS and HNG-X I will revert to the four component
model I used to describe LHITS.
Component A - Counter and Peripherals
4.6.3
4.6.4
The hardware components of HNG-X were almost identical to that for LHITS. I understand
that new Tally roll printers were installed but beyond that the hardware remained largely
the same and the Counter continued to run the Windows NT operating system.
One important change was that every branch received a new router, a piece of hardware
which allowed the branch to connect to the Data Centre. This router had a physical line
® “Introducing Horizon Online’ dated September 2009, page 24 (POLO0086712)..
2° ‘Introducing Horizon Online’ dated September 2009, page 24 (POLO0086712)..
2° ‘Introducing Horizon Online’ dated September 2009, page 24 (POLO0086712)..
46
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
connecting it to the Data Centre but it would automatically switch over to a mobile backup
network, via either Orange or Vodafone (subject to network coverage) if the primary
communication link went down. A related change was that multi-Counter branches were no
longer connected to the Data Centre via the Gateway PC, they connected directly to the
router.
Figure 4.11 HNG-X Counter setup in a multi-counter branch
Multi-counter branch
4.6.5
4.6.6
4.6.7
4.6.8
HN Data Centre
! Communications network (fixed or mobile)
Counter 3 Counter 2 Counter 1
Shared Counter 3 Counter 2 Counter 1 Tintern
weighing peripherals peripherals peripherals A inkjet
scales (keyboard etc.) Mj (keyboard etc.) § (keyboard etc.) aay pees
HING-X's new architecture had four layers: Presentation, Interaction, Business, and Services.
(a) The presentation layer is responsible for displaying information to the user and
accepting user inputs.
(b) The interaction layer provides the foundation for the presentation layers, such as
menus. The combination of the presentation and interaction layers was a replacement
for the Riposte technology.
(c) The business layer provides the business applications in an object-oriented manner.
(d) The services layer provides a set of software objects that support many business
applications. Within the service layer exists a process engine. The process engine
provides a simplified sequence of steps for the counter to deliver services.
Some data is stored persistently at the Counter, such as reference data, process definitions,
and report definitions. Customer transactions are not stored at the Counter.
The service layer is the only layer that communicates with the data centre. The service layer
also provides the interface for on-line services, which includes banking, the use of
debit/credit cards, and mobile phone services.
The biggest change to a Counter Clerk would have been the UI, which changed significantly.
47
Back-office
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 4.12 Changes in the UI between LHITS and HNG-X
4.6.9 In addition, I understand that there were some changes to the available functionality, for
example the ability to transfer sessions between Counters was removed in HNG-X.
Coding of the Counter in HNG-X
4.6.10 HNG-X was coded predominantly in Java, replacing the Visual Basic components used in
LHITS.
Component C - Messaging system
4.6.11 As HNG-X only stored transaction data at the Data Centre, and not locally on the Counter,
the Riposte message store was no longer required. Reference data was still stored locally
on the Counter. The messaging system used the XML format (instead of Attribute Grammars
used in LHITS) and used the TCP/IP protocol (instead of UDP/IP in LHITS) to send data
between the Counter and the Data Centre.
48
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4.7
Component D - HNG-X Data Centres (nee Campuses)
4.6.12
As part of the move to HNG-X there were various updates made to the Data Centre, however
for the purposes of simplicity I will highlight only two. The Branch Access Layer (“BAL”)
and the Branch Database ("BRDB") were new elements introduced in HNG-X. BAL's function
was to exchange messages with the counter software and to perform audit and validation
functions. BRDB was a high-performance Oracle database used to store customer
transactions from all branches.
Updates to Horizon Online
4.6.13
T have only been provided with very limited documentation related to updates made to HNG-
X and therefore the below is by no means a comprehensive list of the changes, let alone
changes that I might consider to be material enough to warrant highlighting.
(a) In 2012, a fix was introduced (known as the “Ping fix”) which was related to Camelot
accounting for the National Lottery. I understand that as part of this fix Transaction
Acknowledgements (“TAs”) were introduced. TAs are non-counter transactions and
typically initiate from somewhere else. This is from another area outside the Horizon
IT System. These transactions are typically relayed to POCL or Fujitsu and need
“accepting” into Horizon before forming part of the branch's transaction data. This
is done by means of TAs sent to each branch. The SPM does not have the option to
reject them.
(b) In 2017 a new version of Horizon was released, HNG-A. I have not been provided
with any substantial documents which detail the changes delivered as part of this
upgrade; however, I understand that one of the major changes was that the
operating system that the Counters used was upgraded from Windows NT to Windows
10. This was necessary due to the obsolescence of Windows NT.
Balancing and Roll-over
4.7.1
4.7.2
POCL procedures required that SPMs undertake various regular processes on the LHITS. One
of the prominent ones I see referenced in the PEAKs, PinICLs and KELs ("PPKs") is in relation
to the Cash Account Period (“CAP”). This was a weekly cycle that started at the start of
business on a Thursday and ran through to close of business on the following Wednesday“?
(in 2004 I understand that POCL moved to a monthly Trading Period ("TP") cycle, with
months made up of 4 or 5 weeks“).
The CAPs are numbered sequentially, (e.g., CAP1 is followed by CAP2, and so on) and mirror
the financial year of POCL which starts at some point in March each year and runs to 52 or
53 weeks. The CAP is of particular interest as it acted as a weekly reconciliation point for a
branch. The data stored in LHITS was compared to the cash and stock physically held in
the branch at the end of the CAP. I understand that this weekly reconciliation process is
referred to as “balancing”, and this was undertaken for each Stock Unit in the branch. Only
Explanation of Local P.O. Reconciliation and Administration, page 8 (FUJ00079193).
TABI, paragraph 241.
49
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4.7.3
4.7.4
4.75
4.7.6
4.7.7
4.7.8
once the SPMs had balanced all of their Stock Units were they permitted to roll-over the
branch to the next week (i.e., the next CAP). The process of balancing their Stock Units and
moving to the next CAP were commonly referred to as “roll-over” or “rolling over”.
A CAP is further divided into Balance Periods (“BPs”). Each CAP must have at least one BP
but can have more than one. BPs allow an SPM to balance their Stock Units without rolling
over to the next CAP. This could be used, for example, to perform an interim mid-week
balancing, or to balance a shared stock unit when one Clerk handed this over to another
Clerk.
The CAP and BP were displayed on the LHITS screen (see Figure 4.8)? and the TP and BP
were displayed on the HNG-X screen (see Figure 4.13).
In order to balance and roll-over an SPM had to undertake various steps, a summary of
which I have reproduced here‘. This process would be undertaken for each Stock Unit that
the branch operated:
(a) check all of the stock (e.g., envelopes etc) they held in branch against the system
held values and adjust these values in the system where required;
(b) declare the stamps that they held in branch (i.e., count up each denomination of
stamps they held and enter these into the LHITS);
(c) declare the cash they held in branch (i.e., count up each denomination of coins and
notes in the till and enter these into the LHITS);
(d) produce the ‘balance snapshot report’ and complete all mandatory checks, making
adjustments to transactions or stock and cash declarations where inconsistencies are
identified, or accepting any discrepancies that LHITS identified between its calculated
values and those from the declarations; and
(e) confirm in LHITS that they wished to roll-over the Stock Unit to the next CAP.
Any loss or gain that was identified through this process must be either posted to the
Suspense Account (pending a correction to the system or an agreement to repay the
amount) or had to be corrected by the SPM adding funds to the till (if a loss) or removing
funds from the till (if a gain)**.
The posting of discrepancies to the Suspense Account was only made once the Stock Unit
had rolled over to the next CAP*S,
Once all Stock Units have been balanced and rolled-over an SPM would produce the Cash
Account report for the branch, which would summarise the position across all Stock Units.
After the IMPACT change was implemented the screen would have shown the TP and BP, as CAP was no longer used.
HSUG, page 627.
HSUG, page 797 to 804.
HSUG, page 516.
50
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4.7.9
4.7.10
4.7.11
4.7.12
4.7.13
4.7.14
The SPM would check this report and if they were happy with this would roll over the branch
to the next CAP.
Whilst this is not intended as a comprehensive list of the tasks an SPM was required to
undertake to roll-over, it is intended to provide an overview of what steps were followed as
part of roll-over and provides the context for the process of checking that receipts and
payments matched.
The Stock Unit balancing process consists of accumulating all the receipts for the Stock Unit
and all the payments for the Stock Unit in the period for which the report is being produced
and ensuring that the total value of receipts matches the total value of the payments. When
this state is reached, the stock unit is said to be 'balanced"**.
I understand the definitions of payments and receipts to be:
(a) Payment: “... a transaction resulting in a payment to the customer (for example,
Alliance & Leicester Giro withdrawals, National Savings withdrawals, Co-op cheque
encashment)”” “Customer Payments”); and
(b) Receipt: *... a transaction resulting in a payment from the customer (for example, an
MVL, TV Licence, Alliance & Leicester Giro deposit, Insurance)” (“Customer
Receipts”).
Ordinarily it is not intuitive that payments and receipts should match one another, however
it is my understanding that the balancing of payments and receipts factored in the cash and
stock balance at the start of a CAP (the so-called “brought forward balance”) as well as
the cash and stock balance at the end of a CAP (the so-called “carried forward balance”).
Payments and Receipts balancing also factors in remittance (remming) activity, revaluations
of stock and internal transfers made between Stock Units.
The balance equations for a Stock Unit are therefore*®:
(a) Receipts = Customer Receipts + Transfers In + Remittances In + Revaluations Up
(b) Payments = Customer Payments + Transfers Out + Remittance Out + Revaluations
Down
(c) Total Receipts = Receipts + Brought forward balance
(d) Total Payments = Payments + Carried forward balance
EPOSS Functional Description, section 11.1 (FUJ00079277).
HSUG, page 358.
HSUG, page 354.
HRR, page 235-236.
51
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
(e) Carried forward balance = Stock on hand + Net discrepancies®?
(f) Total Receipts = Total Payments
* In effect this is the result of undertaking the comparison of physical stock to that held in the system and making any
required adjustments to get these to balance
52
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
EXPG0000001
EXPG0000001
4.7.15 The process of reconciling payments and receipts is perhaps best explained through an
illustration using a simplified and fictional payments and receipt account for a specific Stock
Unit and CAP, in this example Stock Unit AA and CAP15. Note that in this example all
transactions are settled for cash.
Table 4.4 Receipts and Payments Account for AA in CAP15
Brought forward balance from CAP14 into CAP15 for AA:
Cash
‘Stock (stamps)
Receipts
Payment for TV Licence
Payment of road tax
Alliance & Leicester Giro
deposit
Purchase of 20 x 1* class
stamps for cash
Additional money received
("remmed in”) from POCL
£5,000
£500
Receipt Payments
amount
£100
£75
£150
£5
£100
A&L Giro withdrawals
Pension payment
National Savings
withdrawals
Issue of 20 x 1* class
stamps to a customer
Payment amount
£50
£25
£100
£5
Carried forward balance from CAP15 to CAP16 for AA:
Cash
Stock (stamps)
£5,255
£495
4.7.16 There are a few important assumptions I have made as part of this example:
(a) the ‘brought forward balance’ is obtained from the LHITS, based on the agreed
position when the SPM rolled over from CAP14 into CAP15.
(b) the ‘carried forward balance’ is calculated by the LHITS based on the manual cash
and stock declarations made by the SPM at the end of CAP15 and any discrepancies
they accepted as part of the process*,
(c) the payments and receipts are those that have been recorded in the LHITS by the
Clerks; and
s EPOSS Functional Description, page 68 (FUJ00079277).
53
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
4.7.17
4.7.18
(d) as the purchase of stamps by a customer is a purchase of stock, then the item needs
to appear as both a receipt of cash and a payment (representing the value of the
stamps ‘paid’ to the customer).
In this example the total for the Receipts (£5,930) matches that for Payments (£5,930) so
this Stock Unit is balanced and can therefore be rolled-over without the need for further
action.
The format of an actual Stock Unit Balance Report produced by LHITS is somewhat different
as it was, for example, produced on the Tally printer and is therefore a sequential list, rather
than a table.
54
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
5.1
ICL Pathway’s error logging and remediation
Organisational design
BLL
5.1.2
5.1.3
5.1.4
5.15
5.1.6
5.1.7
As part of the incident management process for Legacy Horizon, ICL Pathway utilized a
Fujitsu proprietary call management system for logging errors and defects. The system,
known as PinICL, was created from another ICL custom program with changes made at the
request of ICL Pathway. PinICL was accessible to all Pathway members and used from 1996
to 2003, prior to the PEAK system being introduced.
Internal issues were raised directly into PinICL. These could have been identified during
testing or routine monitoring of the system by ICL Pathway. POCL could also raise incidents
that they themselves found or which could have come from feedback from SPMs. External
users, SPMs, who experienced issues would contact the Horizon Systems Helpdesk ("HSH")
who log the incident via their own dedicated system (“PowerHelp”). If the issue was
deemed necessary for escalation, it would then be recorded in PinICL.
The HSH was the first line of support and were responsible for recording the details of the
incident, diagnosing the problem and attempting to resolve the issue. If the HSH was unable
to resolve the problem, the incident would be routed to a second level support group called
the System Management Centre ("SMC").
The SMC would determine if the incident was a software code problem. If the problem was
a known error, the SMC would determine if there was a workaround recorded in the KELs.
If so, the workaround was communicated to the customer. If no workaround was available,
the SMC ensure there were no duplicate calls for the same problem. If a duplicate call was
identified this incident would be attached to the existing duplicate call. If no duplicate
incident was identified, and the incident was identified to be a software code problem, the
SMC would follow internal procedures and route the call to the System Support Centre
(“SSC”), considered third level support. Hardware issues were generally not routed to the
SSC, although exceptions to this rule exist.
The SSC was responsible for resolving the incidents promoted by the SMC. This was
recorded in the PinICL system. The maintenance of PinICLs was the responsibility of the
SSC through resolution and closure with communications passed back to PowerHelp. If
additional evidence was required, the SMC would be engaged to gather the evidence.
The SSC was tasked with resolving any incidents to the best of their ability and then passing
back the resolution to the SMC. The SSC would triage the incident to determine what other
internal development groups were needed to resolve the incident. Once incidents were
resolved, communications back to the SMC were provided. The SSC also assisted in
maintaining the KELs.
In the event the SSC needed assistance from third party vendors, they escalated calls to
“4th line support” which dealt with technology from outside suppliers. The “4th line support”
was also responsible to the SSC in terms of updating PinICLs and entering resolutions into
55
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
5.2
5.3
the KEL database. They ensured fixes or workarounds have been tested prior to passing to
the SSC.
PinICLs and PEAKs
5.2.1
5.2.2
5.2.3
5.2.4
5.2.5
KELs
5.3.1
5.3.2,
Each incident logged in the PinICL system is referred to as a PinICL. As noted above,
sometime in 2003 ICL Pathway began using the PEAK system for incident management and
thereafter each logged incident was referred to as a PEAK. There appears to be no significant
difference in content between a PinICL and a PEAK. The only difference is the PEAK system
was a web-based system and utilized Hypertext Markup Language (HTML).
As new PPs are logged by a Team Member ("TM") they are assigned a unique reference
number. Additional attributes are captured such as who logged the PP, when it was opened,
the last update date, open/closed status, summary of the issue, and product group. If
available, information is captured relating to work packages, fixes, other PPs to reference, etc.
The body of a PP is chronological and typically begins with the TM describing the issue that
was identified, assigning a call priority, call type, estimated completion date, and routing to
a Team Leader ("TL"). The TL reviews the call, provides approval or rejection (return to TM
for further action or close if not valid) then routes the call back to the relevant TMs as defined
by the products being impacted.
When a TM received a PP, they attempted to diagnose the error and identify a fix. If more
information was required, they would request additional evidence when needed to recreate
the incident. TMs also checked KELs to ensure that the incident was not already addressed.
Calls were passed between TMs as they diagnosed the issue and attempted to resolve it. Often
this would be represented with an update to a Response Category recorded within the PP.
ICL Pathway and Fujitsu maintained a knowledge base of information that included known
issues in the Horizon IT System. This knowledge base was referred to as the Known Error
Log ("KEL"). Individual entries are referred to as KELs. KELs contain information on how
to address or rectify issues that have previously been identified within the Horizon IT
System.
KEL maintenance is the responsibility of both the SSC and "4" line support.” They can be
referenced during the resolution of a PP. They contain structured attributes such as type,
summary, open/closed date, status, and visibility. The body of a KEL contains information
covering the symptoms, problems, solutions, and related evidence.
56
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
6.1
Materials provided to me
Overview
6.1.1
6.1.2
6.1.3
6.1.4
6.1.5
6.1.6
This section of the Report outlines the documents and data (collectively referred to as
“Materials”) that I was provided. These materials were provided to me by the Inquiry.
I characterize the Materials into two categories:
(a) Primary Materials: These relate directly to the period up to and including the Roll Out
of the LHITS; and
(b) Background Materials: These provide both background on the LHITS system and
processes and procedures.
In summary, the Primary Materials include:
(a) Extracted IT incident tickets ("PinICL(s)") from Fujitsu's original proprietary call
management system (“the PinICL System”);
(b) Extracted IT incident tickets ("PEAK(s)”) from Fujitsu's successor proprietary call
management system (“the PEAK System”);
(c) Two archived PinICL databases (in Microsoft Access format) (the “PinICL archive
databases”);
(d) Extracted records ("KEL(s)") from Fujitsu's knowledge management tool ("the KEL
System”); and
(e) Acollection of monthly reports prepared by ICLPL in relation to the development and
roll-out of the Horizon IT System (“the Monthly Reports").
I understand that documents from the PinICL System, the PEAK System, the KEL System,
and the Monthly Reports were produced by Fujitsu in response to the request submitted to
them by the Inquiry.
The main Background Materials I relied on are those documents referred to in section 4.1.3.
In addition, the Inquiry directed me towards the following publicly available documents as
further Background Materials:
(a) The ‘Terms of Reference (updated)' for the Post Office Horizon IT Inquiry;
(b) The ‘Completed List of Issues’ for the Post Office Horizon IT Inquiry;
(c) Bates & Ors v Post Office Ltd (No.3) "Common Issues") [2019] EWHC 606 (QB) (15
March 2019) (Common Issues Judgment);
57
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
(d) Bates & Ors v the Post Office Ltd (No 6: Horizon Issues) (Rev 1) [2019] EWHC 3408
(QB) (16 December 2019) (Horizon Issues Judgment);
(e) The Horizon Issues Judgment Technical Appendix,
(f) The Horizon Issues Judgment Appendix 2 - ‘Summary of Bugs, Errors, Defects’; and
(g) The Horizon Issues Judgment Appendix 3 - ‘Glossary’.
6.1.7 An inventory of all of the documents I relied on in producing this Report is provided in
Appendix A.
6.2 PinICLs and PEAKs
PinICLs
6.2.1. The PinICL System was the customised incident logging and resolution tracking system
adopted for use by ICLPL to support the Horizon IT System during the period 1996 to 2003,
prior to the introduction of the PEAK System in 2003°.
6.2.2 Each ticket logged on the PinICL System is referred to as a “PinICL" within my Report.
6.2.3. PinICLs recorded (amongst other things) incidents:
(a) identified by ICL Pathway in test systems;
(b) identified by ICL Pathway during routine system monitoring;
(c) raised by POCL, some of which may have resulted from feedback received from SPMs;
and
(d) resulting from certain calls made to the Horizon System Helpdesk by SPMs, which
were deemed necessary for escalation through ICL Pathway’s incident management
process.
6.2.4 The standard format of a PinICL is divided into four major sections:
(a) the header which contained summary level information;
(b) the reference table, which contained data points such as customer reference, fast
track, and work numbers;
(c) the products table, which contained product groups, product names, and product
versions; and
(d) the activities log, which contained the running commentary about the PinICL.
Submissions on behalf of Fujitsu Services Limited dated 13 September 2022 (in response to a Rule 9 Request dated
29 April 2022) (FUJO0119556).
58
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
6.2.5 An example PinICL is included in Appendix C.
6.2.6 There were 56,489 PinICLs produced to me, and all were provided as PDFs.* The PinICLs
provided were opened between 7 July 1996 and 31 December 2000. I understand that
Fujitsu chose the upper date limit (31 December 2000) when deciding which documents
were responsive to the Inquiry’s Rule 9 request. I understand that the PinICL System
allowed a user to attach supporting documents to each PinICL, however these supporting
documents were not readily available and therefore, they were not included as part of the
production by Fujitsu.
PEAKs
6.2.7. In 2003, ICLPL replaced the PinICL System with the PEAK System. The PinICL System was
archived and open tickets from the PinICL System were migrated (transferred) to the PEAK
System.
6.2.8 Each ticket logged on the PEAK System is referred to as a “PEAK” in my Report.
6.2.9 __ It is my understanding that the PEAK System served a similar, if not identical purpose, to
the PinICL System and therefore the origins of the tickets within it would be much the same
as those identified above for the PinICL System. As the function and content of the PinICLs
and PEAKs are the same I will refer to these collectively in this Report as “PP(s)”.
6.2.10 The standard format of a PEAK is similar in layout to a PinICL with the main exception being
the layout of the activities log.
6.2.11 An example PEAK is included in Appendix C.
6.2.12 There were 16,530 PEAK related documents produced in various file formats (HTM, DOC,
XLS, BMP, and TXT)°5. I understand that the PEAK System allowed a user to attach
supporting documents to each PEAK. These documents were produced by Fujitsu. This
document population represented 13,442 PEAKs and 3,088 supporting documents.
6.2.13 The PEAKs provided were opened between 29 May 1997 and 31 December 2000.
Replacement PinICLs and duplicate PinICL/PEAKs
6.2.14 During the initial review of the PinICLs two observations were made:
(a) Anissue with the ordering of the ‘Activities’ table® was identified. This issue meant
that it was not possible to easily read the entries within the comments field.
(b) The 13,442 PEAKs were duplicated in the PinICL population.
Portable document format.
‘Submissions on behalf of Fujitsu Services Limited dated 13 September 2022 (in response to a Rule 9 Request dated
29 April 2022) (FUJO0119556).
These extensions are: Hypertext markup, Microsoft Word, Microsoft Excel, Bitmap image file, and a text file
respectively.
A table in each PinICL that provides a sequential log of activities recorded in relation to customer calls.
59
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
6.2.15
6.2.16
These observations were raised with the Inquiry who then communicated them to Fujitsu.
Fujitsu responded that:
(a) They had identified 17,537 PinICLs with the ordering issue and subsequently
reproduced 17,537 PinICLs.
(b) They recommended I use the PEAK version’.
None of the 17,537 Replacement PinICLs had duplicate PEAKs versions (i.e., these two
document sets were mutually exclusive).
PinICL archive databases
6.2.17
6.2.18
6.2.19
6.2.20
In response to queries raised with Fujitsu, two archive databases (in Microsoft Access
format) were produced:
(a) ‘PinArchive1’, being a .MDB file; and
(b) ‘PinArchive2', being a .MDB file.
Collectively these two databases are referred to as the “PinICL archive databases”.
PinArchive1 contained 12 different data tables, while PinArchive2 contained 127 different
data tables. Fujitsu confirmed that both databases were archives from the PinICL system,
these standalone archives do not share PinICL reference numbers with each other (i.e., the
PinICL records they contain are mutually exclusive), and the PDF PinICL documents provided
were derived from these databases.
The PinICL archive databases were received late in my review. I therefore decided, in
consultation with the Inquiry team, not to fully investigate the databases as this would have
unduly delayed the completion of this Report, which could have had knock-on consequences
for the Inquiry’s timetable. In addition, I noted that Fujitsu had not produced these PinICL
archive databases in response to the original Rule 9 request submitted by the Inquiry in
December 2021. Therefore, I deduced the incremental information not to be responsive to
the original request from the Inquiry.
Summary of the PP data used to undertake my review
6.2.21
The dataset changed over the course of the review as I received multiple copies of the same
(or very similar) data across different deliveries. I therefore had to make decisions as to
what datasets to use. I also had two analysis workstreams, which were at different states
of progression when some of the additional data was provided. I decided that these
workstreams should, in some cases, use different datasets. The two workstreams were:
Fujitsu cited two reasons for this recommendation - (a) PEAK is a live database and will therefore capture any updates
to the equivalent PinICL record after it was migrated; and (b) Records extracted from the PEAK database also contain
their attachments as family member documents, where available. No such attachments are readily available in
relation to records held in the PinICL archive.
60
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
(a) Analytics: This workstream was focused on structuring the PP data using the
Microsoft Azure services so it could be analysed en masse for the entire PP population.
(b) Document Review: This workstream was focused on using industry standard
document review and machine-learning tools, in conjunction with manual document
reviews, to analyse the unstructured components of the data (i.e., the human
generated comments) to identify themes within the PP population.
6.2.22 This table summarises the PP documents and data that I relied upon as part of my review:
Table 6.1 PP documents used in the review
Document Production _ File Reference and Total Used for Used for
type format description documents Analytics? Document
(files/ Review?
records)
PinICL ticket First PinICL .PDF Al. Incorrectly 17,537 No No
production ordered PinICLs
-PDF A2. Correctly 13,442 No No
ordered PinICLs
(duplicates)
.POF A3. Correctly 25,510 No Yes
ordered PinICLs
(non-duplicates)
Second -POF A4. Replacement 17,537 No Yes
PinICL PinICLs
production
Third PinICL — .MDB AS. Microsoft 56,489°° Yes No
production Access database
PinICLs
PEAK ticket First PEAK -HTM B1. PEAK - Tickets 13,442 No Yes
production
PEAK First PEAK -Doc B2. PEAK - 3,088 No Yes
attachment —_ production XLS Attachments
-BMP
xT
6.2.23 Neither the Analytics nor the Document Review workstreams used the following documents
sets:
(a) A1, as it was agreed with Fujitsu that these contained errors and were therefore
replaced by document set A4;
(b) A2, as it was recommended by Fujitsu that I use the PEAK versions of these,
contained in dataset B1.
6.2.24 The Analytics workstream solely relied on the PinICL data that was obtained from the Archive
PinICL databases (AS), as the objective of this workstream was to structure the data, as
5° This database also contained PinICLs that were opened on or after 01 January 2001.
61
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
best as possible, before analysing it. The Archive PinICL databases were better structured
than the PDF PinICL documents provided, so it was decided to use the databases for this
workstream. Owing to the non-standard format of the PEAK attachments these were not
used in the Analytics workstream.
6.2.25 Upon analysis of the contents of the PinICL archive databases it was discovered that all
13,442 of the PEAKs were also contained in them, which is consistent with our understanding
that these were open PinICLs at the point of migration to the PEAK System. It was decided
to use the data from the PinICL archive database for the PEAKs to undertake quantitative
analysis, as this was in a readily interrogable format’?
6.2.26 The Document Review workstream was already well advanced by the time the Archive PinICL
databases had been received and analysed. In addition, this workstream required data files
to be produced from the Analytics workstream to consolidate the searchable text (principally
the ‘Comments’) into a single text string. For these reasons I decided that the Document
Review workstream should continue to use the PDF PinICL documents (datasets A3 and A4)
and the HTM PEAK documents (dataset B1) as its data source. In addition, the PEAK
attachments (dataset 82) were made available to the reviewers in the Relativity platform.
6.3 Known Error Logs (“KELs”)
6.3.1 The “Known Error Log” was a knowledge management tool used by both ICLPL and Fujitsu
to explain how to deal with, or work around, issues that arose in the Horizon IT System.
6.3.2 It is my understanding that the KEL system was available (in read-only mode) to the
following support teams: the Horizon System Helpdesk (“HSH”) and the System
Management Centre (“SMC”). Only the System Support Centre ("SSC") team were
permitted to create new KELs and update existing ones.
6.3.3. The structure of a typical KEL contains the following sections:
(a) Header: This section contains the meta data for the KEL including the name of the
Fujitsu employee who raised it, identifier for the PEAK or PinICL that originated the
KEL, version number of the KEL, etc.
(b) Symptoms: This section describes the issues experienced by the SPM in the Horizon
IT system.
(c) Problem: This section describes the underlying cause for the symptoms experienced,
as diagnosed by the SMC or SSC. This would also be reflected in the underlying PEAK
or PinICL.
(d) Solution: This section explains how to deal with, or work around, the issues that
arose in the Horizon IT System.
5° Checks were performed to compare equivalent records in the PinICL archive database against those from the HTM PEAKs
documents.
62
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
6.4
6.3.4
6.3.5
6.3.6
6.3.7
6.3.8
(e) Evidence: This section generally lists the log files reviewed to investigate the issue(s)
and provide the solution(s).
I note that not all sections are completed in all of the KELs, indicating that not all of the
sections were mandatory when creating or updating a KEL. The typical naming convention
for the KELs is as follows: <Initials of first name of the Fujitsu employee who raised the
KEL><Last name of the Fujitsu employee who raised the KEL><3-4 numbers><A letter>
e.g., “SParker538M". A KEL can have multiple versions during its lifecycle. For example, KEL
“reoleman1253{” appears to have two versions.
An example KEL is provided in Appendix C.
The term Known Error Log or KEL was replaced in around July 2019 by the term "Knowledge
Base” or “KB”.
There were 656 KELs produced in HTM format.
The KELs provided were opened between the dates of 26 May 1998 and 28 December 2000.
Management Reports (“MRs”)
6.4.1
105 management reporting documents were produced.
(a) 19 ‘Pathway Programme Monthly Reports’: These documents summarize the business
development activities of the Pathway Programme.
(b) 13 ‘Monthly Joint Implementation Reports’: These documents are implementation
reports jointly issued by ICL Pathway and POCL.
(c) 4 “ICL Pathway Customer Service Reports’: These documents contain summaries of
the performance of the ICL Pathway Customer Service Business Support Unit.
(d) 44 ‘Pathway Monthly Reports’: These documents are comprehensive management
reports for ICL Pathway ranging from October 1996 through December 2000. I was
not provided reports for every month in this time range. Two Managing Directors
were the approval authorities for these reports. J. H. Bennett was the approval
authority up to November 1999; M. Stares was the approval authority beginning in
January 2000. They typically cover the following areas:
(i) Managing Director's Summary
(ii) Systems Report
(iii) Commercial and Financial Report
(iv) I Customer Requirements Report
(v) Customer Service Report
63
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
6.4.2
(vi) Quality and Risk Report
(vii) Business Development Report
(viii) International Sales Report
(ix) Organisation & Personnel Report
(x) Post Office Client Report
(e) 17 ‘Monthly Reports’: These documents are short reports related to ICL Pathway.
(f) 8 ‘Monthly Performance Reports’: These documents are short reports related to ICL
Pathway performance.
Based on the richness of content, my primary focus was on the Pathway Monthly Reports.
Four of these reports appeared duplicative of other reports in the set; consequently, forty
Pathway Monthly Reports were in my final review set.
64
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
PART 2:
My review and observations
65
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
We Pre-proces:
g of documents
7A Overview
wad
This section describes how the source data I received was processed, in preparation for it to
be analysed.
7.2 Analytics workstream
PEAKS and PinICLs
7.21
7.2.2
7.23
7.24
7.2.5
7.2.6
7.27
A DAT file (a structured text file) was provided that served as a manifest for the delivery of
PPs and KELs. The original file name acted as the reference number. Additional metadata
fields including a custodian field which indicated the report type (e.g., PEAK, PinICL, or KEL).
The DAT file also provided parent child relationships which associated supporting documents
to their PP.
The PPs were processed through Microsoft Azure's Form Recognizer ("Form Recognizer”)
service to organize the components of these reports for further analysis.
Form Recognizer is an Artificial Intelligence ("AI") service that applies advanced machine
learning to transform unstructured documents into actionable datapoints/datasets by
extracting text, key value pairs, tables, and structures from documents.
Form recognizer can accept many different file formats; since the largest portion of delivered
documents (PinICLs) were PDFs, it was decided to standardize the PEAKs (and KELs) into
PDFs. The PEAKs and KELs were in HTM format.
The transformation of PEAKs and KELs was accomplished through a series of Python libraries
that rendered the HTM files into PDFs.
Once form recognizer processed a document, the OCR® text, key value pairs, table
structures, and named entities were returned in a structured text format known as a JSON
file. One JSON file was returned for every document put through the form recognizer. These
JSON files were then ingested into a Microsoft Azure Databricks repository for further
analysis.
This process was successful for 57,137 documents. Seven documents could not be
processed, despite multiple attempts to reformat the files. They were omitted from further
analysis.
© Optical character recognition
66
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
7.3
Document Review workstream
Introduction - Software used to analyse the PPs
7.31
Owing to the number of PPs to be reviewed and their esoteric nature, it was quickly
determined that a linear review would be impractical and inefficient. The following industry-
standard software platforms were used for the analysis and review of PPs:
(a) __ Relativity is a market leading document review platform used to display metadata
and enable searching and review of documents in an efficient and audited manner.
Relativity allows for multiple people to look at documents concurrently and to save
decisions in coding fields to classify and group documents accordingly.
(b) Brainspace is an advanced data analytics platform available for investigations,
eDiscovery, intelligence mining, and compliance reviews. Brainspace uses machine
learning technology to provide information on a set of documents.
PPs - Document Review Setup
7.3.2
7.3.3
7.3.4
735
The documents and the related DAT file were loaded into Relativity for searching and review.
Further fields which had not been provided in the DAT file were extracted from the PPs by
the Analytics workstream. Additionally, the primary content of the PPs were processed for
consumption by Brainspace. This was necessary because the original format of the PPs was
not an optimal format for Brainspace to analyse.
As mentioned above, there were seven PinICLs which were not included in the Brainspace
build as their text extraction caused errors.
In order to facilitate the review of the PPs, a set of Response Categories and Defect Causes
were extracted from the following source documents, as described in correspondence
received from Fujitsu:
(a) Section 15 of the ‘PinICL Incident Management Process’, dated 30 January 1998°
(b) Section 8 of the ‘PinICL User Guide’, dated 15 February 2000%
Searches were run to identify instances of Response Categories and Defect Causes within
documents which were then highlighted in the Relativity document review screen so that
they could be easily identified when undertaking manual review, as illustrated here:
Submissions on behalf of Fujitsu Services Limited dated 13 September 2022 (in response to a Rule 9 Request dated
29 April 2022) (FUJO0119556).
PinICL Incident Management Process, Section 15 (FUJ00098253)
PinICL User Guide, section 8 (FUJ00098255)
67
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 7.1 Example of highlighted codes in Relativity
Responded to call type B as Category 30 -TL confirmed
The response was delivered on the system
The Call record has been transferred to the Team: QFP
Hours spent since call received: 0.3 hours
Target Release updated to IR - NR2
F} Response
PLease investigate
[END OF REFERENCE 8068675]
Responded to call type B as Category 38 -Potential Problem Identified
KELs - Document Review Setup
7.3.6 As with the Ps, the raw data provided by Fujitsu was loaded into Relativity for searching.
Based on an initial review of the KEL documents it was decided that analysis in Brainspace
would not yield meaningful insight. Relativity was used to undertake a review of the KELs.
Supervised Learning using Brainspace
7.3.7 Supervised learning (a type of machine learning) is a process through which Brainspace is
provided with examples of relevant and non-relevant documents. Using those examples,
the system identifies other documents which are conceptually similar and may also be
considered "relevant." This process involves the following steps:
(a) Seed set documents are identified, to include both positive and negative examples of
an issue, usually referred to as “Relevant” and “Not Relevant” documents
respectively.
(b) A Continuous Multi Modal Learning (*CMML") model is set up within an existing
dataset, using the seed set documents which are compared against the rest of the
population.
(c) The CMML model assigns a relevance rank to all documents which can be analysed
within the population, where the scores depict the following:
(i) Relevance rank 0.0 to 0.4: Documents are likely to be Not Relevant;
(ii) Relevance rank 0.4 to 0.6: The model requires more information to make a
decision on these documents; this is known as the “uncertain zone”;
(iii) Relevance rank 0.6 to 0.8: Documents are likely to be Relevant; and
(iv) Relevance rank 0.8 to 1.0: Documents are very likely to be Relevant.
(d) The CMML model continues to be trained through further review of documents and
identification of both positive and negative examples
68
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
7.3.8 The model also tracks consistency of review, to identify examples of documents which are
inconsistent between how the model believes it should be tagged, and how a reviewer has
tagged it.
Keyword searches using Relativity
7.3.9 Inorder for the PPs to be searched with keywords, the Extracted Text field (that is, the body
text of the PP) was added into an index using dtSearch; an industry standard search engine
used by Relativity.
7.3.10 Specific keywords were selected to target PPs which were likely to provide examples of a
theme, such as searching for the following keywords to identify PPs relating to connection
issues experienced:
(a) (network OR isdn OR polling) AND (issue OR error OR failure)
7.311 PPs returned by these keywords were subsequently reviewed and coded where a good
example was identified.
7.4 Review Approach
Monthly Reports
7.4.1 I reviewed these documents using a manual process. This manual process consisted of
reading each document, assessing the information in the document, noting sections that I
wanted to revisit, and then organising these notes for further review. I then iterated through
the documents again to refine my notes that eventually resulted in the themes and
observations documented later in this Report.
PinICLs and PEAKs (PPs)
7.4.2. PPs that were identified for review (either by Brainspace’s Supervised Learning or Relativity
key word searches) were manually reviewed in Relativity. Each PP would be classified as
‘Relevant’ or ‘Not relevant’ depending on the particular theme that was being investigated.
KELs
7.4.3. Inorder to focus the review of the KELs, an analysis of the PPs was performed. This analysis
used a Regular Expression (regex) pattern to identify KEL references within the PPs. A regex
is a sequence of characters that specifies a search pattern in text.
7.4.4 This resulted in 1,380 KELs identified in the text of the PPs, only 332 of these were contained
within the KELs produced to me.
7.4.5 In response to my question on the missing KELs, Fujitsu explained, in their correspondence
dated 12 August 2022, that some KELs may have been deleted and therefore only the
recoverable KELs for the Relevant Period®* were provided.
® Period on or before 31 December 2000
69
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
746
TAT
The 332 KELs were manually reviewed in Relativity and were classified in two ways according
to the element of LHITS that they related to:
(a) _ Responsiveness: This is a classification of whether the KEL related to a recognised
issue in any part(s) of the LHITS. The two different options were:
(i)
(ii)
Responsive: If the KEL did relate to a recognised issue in any part(s) of the
LHITS then the KEL was classified as “Responsive”;
Non-responsive: If the KEL did not relate to an issue with LHITS then the
KEL was classified as “Non-responsive”. The scenarios identified were as
follows: the KEL related to identified human errors by branch staff; the KEL
related to a clarification of the exact process required to correctly operate the
Counter system; or it was not possible to determine from the text of the KEL
what the KEL related to.
(b) Nature of the KEL: For responsive KELs this is a classification of the part of LHITS
that the KEL related to, or for non-responsive KELs it was the reason the KEL was
determined to be non-responsive. The different options for responsive KELs were:
(i)
(ii)
(ii)
(iv)
vy)
(vi)
(vii)
(viii)
(ix)
Hardware: A recognised issue with the hardware components of LHITS;
Software: A recognised issue with the software components of LHITS;
Download / Synchronisation / Roll-out: A recognised issue with the
synchronisation components of LHITS that exchanges data or updates
between the different components of LHITS;
Communications / Network: A recognised issue in the internet connection
or a system communication issue that permits the different components to
communicate with each other;
Central servers: A recognised issue with the central server components of
LHITS;
Process related: A recognised issue in the standard processes that support
the correct running and operation of the LHITS (e.g., distributing software
updates, distributing reference data updates);
Operating System / Disk full: Issues with the OS or memory issues on the
Counter;
Corrupt message store: Issues with the message store corrupting; and
Other: An issue not falling into one of the above categories
Based on the review it was determined that, for some KELs, multiple ‘Nature of the KEL’
classifications were appropriate as the KEL related to more than one component of the LHITS.
70
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
748 Where relevant I have included the results of the KEL review in the overall themes I
identified.
71
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
8.
6
SPM tra
8.1.1
8.1.2
8.1.3
8.1.4
g experienced difficulties during National Rollout
As noted in my theory section, systems raison d’etre is to serve the enterprise's business
processes. An important aspect of this endeavour is that the users of these systems
understand how they operate. Training is the first step of this educational process. It is
apparent from reading the ICL Monthly Reports that there were significant problems in
training the SPMs as they adopted the Horizon IT System.
In addition to the challenge of training users on the Horizon IT System, there was also a
challenge of training users on computers in general, as acknowledged publicly by Fujitsu in
their case study for the Horizon IT System®°:
“Before Horizon can be installed, a great deal of groundwork has to take
place. Each Post Office branch is surveyed and prepared, with the new
electrical cabling and counter space being installed where necessary.
Counter staff receive a day's training and office managers and
subpostmasters attend a one-and-a-half day course, delivered around the
country. At the height of automation, over 300 branches were automated
per week. Training was provided to 63,000 staff members from the age of
16 to 87 with various skills levels (this number includes 2,000 staff
members who were over 80 years old). Approximately five thousand calls
were received each week by the Helpdesk, due to the Counter staffs! lack
of computer experience.” (emphasis added).
As illustrated in the following table, ICL Pathway was aware of the importance of training
the SPMs. They noted early (April 1999) in the national rollout that SPMs were facing
difficulties in moving from a paper based balancing process to the automated balancing
process resident in the Horizon IT System. To address this situation, ICL Pathway
emphasized the importance of increasing the training available to SPMs. However, as the
summer months proceeded, these balancing issues persisted. By the autumn of 1999, a
joint report issued by ICL Pathway and POCL acknowledged that training continued to cause
major difficulties. These difficulties continued into 2000 resulting in ICL Pathway believing
that POCL was so dissatisfied with training (among other issues) that POCL would pursue
commercial remedies.
The following tables contain verbatim extracts from the monthly reports (MRs) and PinICLs
and PEAKs (PPs) which I relied on in identifying this theme. I have intentionally not made
any corrections to grammar or spelling. Where I deemed it helpful, I have highlighted certain
sections in bold. The views expressed in these extracts are that of the authors, being
principally ICL Pathway, but in some cases ICL Pathway and POCL.
Fujitsu case study: https://www.fujitsu.com/downloads/SVC/fs/casestudies/uk-postoffice2. pdf
72
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Table 8.1 Verbatim extracts from Monthly Reports
URN
FUJ00058181
Fuj00058183
FUJ00058186
FUJO0058186
Title
ICL Pathway
Monthly Report -
April 1999
ICL Pathway
Monthly Report -
June 1999
ICL Pathway
Monthly Report -
September 1999
ICL Pathway
Monthly Report -
September 1999
Date
April-99
June-99
September-
99
September-
99
Extracted Text
In the first 4 weeks of live NR2 service, it has become
evident-that postmasters have been experiencing
difficulty managing the change from a manual
balancing process to automated balancing. To address
this concern improvements in training have been made
to put a greater emphasis on practical experience in
balancing. HFSOs, supporting first office balances,
have received a refresher course with the focus being
on balancing. The Sub-postmasters managers course
has been extended to two days, with the extra half day
being used to provide additional time on the topic of
balancing and practical experience-in the balancing
process.
From Pathway's perspective, CSR (LT1) has continued
to perform reliably. POCL's perception is dominated by
continuing end-user problems with stock balancing and
cash account production on Wednesdays. Although
many software fixes have been applied to LT1, there
remain several outstanding that will not be
implemented until LT2. The majority of problems relate
to:
1. Payments not equal to Receipts
2. Printing/printer performance
3. Effectiveness of Training
Although National Roll out rates have risen to 200 Post
Offices per week, the level of issues occurring on
installation day and the level of training scheduling
failures puts achievement of the 300 offices per week
roll-out rate required in 2000 at risk. Knowledgepool
are introducing new scheduling software and a plan of
activity to remove/reduce the causes of the other
issues is being put in place for the November to
January break in National Roll-out.
There is currently a serious issue relating to the
scheduling of training events within the
Implementation programme. The training scheduling
system of Pathway's training sub-contractor,
Knowledgepool, has been struggling to cope during the
early part of national rollout, although a planned
system replacement was imminent. During September
the training scheduling system crashed resulting in a
loss of data and some data corruption. The new
system was introduced over the weekend 2/3 October,
with some teething troubles. Recent training
scheduling failures (late training invites. or no training
prior to installation in a small number of cases) were
caused from the data loss and data corruption of-the
original system. Manual checks have, been
implemented to minimise further disruption and the
benefits of the superior replacement system will be
available for future training scheduling, although the
main benefits will only be seen after the Xmas break.
73
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
NFSPOOO00065
FUJ00058191
FUJ00058191
FUJ00058197
8.1.5
Table 8..
URN
Fuj00029755
Title
ICL and Post
Office Monthly
Joint
Implementation
Report covering
27 September
1999 to 24
October 1999
ICL Pathway
Monthly Report -
May 2000
ICL Pathway
Monthly Report -
May 2000
ICL Pathway
Monthly Report -
December 2000
Date
27
September
1999 to 24
October
1999
May-00
May-00
December-
00
Extracted Text
Training continues to cause major difficulties. A variety
of different issues have been encountered including,
outlets not contacted to book training, outlets turning
up to non-existent courses, outlet staff being booked
onto wrong course type. This is compounded by the
fact that the daily & weekly reports are not received at
the scheduled times. This is further compounded by
the reports being inaccurate. KPL have also been
unable to respond to issue raised by the TLM in a
timely fashion, or occasionally at all.
POCL perception of SLAs and Training, and also of our
commercial attitude to risk taking on new business: all
negative as epitomised by the recent Dave Miller
letter. Hopefully the away day will improve that
perception: Risk remains that POCL will extract
commercial concessions out of us (meaning
unbudgeted cost).
POCL are shaping up to hit us on SLAs and Training.
This was predicted for about now on the basis that, in
the case of help desk metrics, we will have failed to
meet all criteria for three successive quarters. That
gives POCL the right to terminate the contract. We
don't expect them to want to do that, but they can be
expected to use the ‘default’ as a lever to force us to
do better and make concessions.
A settlement for the projected shortfall in training
courses against the contracted number, arising from
low course occupancy levels, has been agreed with the
Post Office. As part of a package to achieve relaxations
against existing service SLAs, Pathway will pay the
first £1M of the training shortfall. Beyond this PON and
Pathway will share the shortfall equally. Measures to
improve occupancy levels have been implemented and
consequently reductions in the estimated shortfall
have been achieved in each of the last three months.
Initial occupancy levels in January are also favourable.
The cost of the projected shortfall has therefore fallen
from £1.3M to £1M. Efforts continue to improve this
with the aim of reducing Pathway’s contribution. This
improvement however represents a £300K saving
compared to last month's financial forecast.
A review of the PPs reinforces the theme that the SPMs were reporting that the lack of
training was problematic in their execution of business activities. Additionally, SSC staff
were also raising concerns about the ineffectual nature of training. In these examples, I
have emboldened some sections of each entry, but have included wider passages for
context.
2 Verbatim extracts from PPs
Ticket Source
PinICL
Date
24/09/1999
Extracted Text
"I do not think that the documentation covers
this type of transaction and there is no mention
of it in the training manual... I have spoken to
Audrey Adams and she will liaise with POCL and if
necessary raise a note for distribution to POs."
74
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN Ticket Source
FUJ00032293 PinICL
FUJ00030982 PinICL
FUJO0031101 PinICL
“ Horizon Field Service Officer
Date
19/10/1999
23/10/1999
24/10/1999
EXPG0000001
EXPG0000001
Extracted Text
"All SU's are apparently in CAP31 at present. I have
agreed with the PM to try and arrange for HFSO
Andrew Perkins to visit the site next week to try and
resolve the various issues the PM has. Will call PM
back later today to try and confirm that arrangements
have been made." "NBSC have stated there are
no HFSOs available to help this PM. At present he
does not have enough knowledge of the system
for SSC/HSH to advise him. He requires onsite
training and until this is provided by POCL SSC
are unable to help him. This is not a software issue, it
is a training issue and the PM is aware of this. 1
have spoken to the PM and he has agreed to fax his
last CAFinal report to us." "PM is not happy with the
service he is receiving. He has not heard from
anyone and it will soon be Wednesday again. He
advised that it is so frustrating when no-one tells you
the answer. PLEASE CAN PM BE CONTACTED." "I have
looked at the message store for this FAD, the
problems mainly arise from use of the suspense
account over the last 4 or 5 weeks. This is not a
software issue and as such should be dealt with by
POCL, in particular, an HFSO needs to visit the site
asap. I have voiced Julie Welch about these
problems."
"Looked at outstanding call "9910010196' - the PO still
has an outstanding descrepancy of £47,000 - which
the HFSO and SSC has been investigating.” "PM
very unhappy with situation - stated she has had this
problem for approx three weeks. Is not satisfied as she
was advised to call back today - and the problem is
still unresolved. Reluctantly agreed to wait until the
HFSO is arranged. - previous HFSO was S. Warwick."
"On checking open calls troubleshoot it appears that
this PM has problems each week with balancing. Is
there a system problem or a training issue. Please
investigate.” "The original problem with zeros on
the trial balance and balance shapshot is described in
KEL "All entries on report are zero". This will have
been corrected by now. The other problems reported
on this call appear to have been copied from other
calls and will be dealt with under their original call. If
the PM is having big problems each week, then
yes, we would agree that there is a training
problem here. Especially since the PM appears to be
requesting an HFSO, that will be the best way forward.
The Pm has not been contacted.”
"No fault in product. The system is working as
designed. The PM has declared his cash as a loss, and
posted this to the suspense account as a loss, these
are both for £2.52 giving a net of £5.02. PM is not
understanding how the suspense account works. PM
needs to be advised on how the suspense account
works. This is not a software fault. PM not
contacted. Closing call as no fault in product.
Training issue"
75
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
Fuj00045829
Fuj00040054
FUJ00066611
8.1.6
8.1.7
Ticket Source
PinICL
PinICL
PEAK
Date
26/11/1999
30/03/2000
30/08/2000
Extracted Text
"Please confirm that there will be some sort of training
in the outlets to warn PO clerks about the change in
trans id format & we will happily close this pinicl as no
fault in product” "Training is one-hit only, i.e.
staff are trained by Pathway on ONE release of
Horizon only, CSR or CSR+. Therefore, unless
POCL specifically require us to do "backfi
training, CSR trained staff are not retrained on
CSR+, so there is no switchover training issue to
consider. At CSR+ we will train on the new
transaction id formats. "This is not the
answer we expected. Is there no way
(Memoview for example) in which PO staff can
be advised of such changes? Should this be chased
or not?" "Given the lack of progress to resolve
this issue it is suggested that it becomes a problem
that requires a design statement to be made...As such
it will be assigned as a design problem for
documentation and then if required softwre
resolution.”
"Call raised to look at issues at this site as PM believe
there are software problems.” "Information:
Update from Peritas: PM has had system problems for
several weeks. system seems to alter figures at
random. having taken advice, I told PM that I have to
pass the call over to systems staff, as all payment,
reciepts and reports were correct. Please investigate."
" As per telecon with Gary @ NBSC this call is being
transferred to SSC for investigation." "In all
cases Payments and Receipts match. As I
suggested on an earlier call for this PO, I believe
that the PM is in need of training, to understand
how the balancing process works.”
"He feels that he has not received sufficient
training, and admits that if he was trained properly,
he may be able to get through balancing a bit quicker.
The PM has requested additional training, which was
granted, but his RNM cancelled it without letting him
know, then denied cancelling it??? The PM seems to
have an issue with the RNM, in that he feels that he is
not helping him resolve any issues." "Have escalated
the PM's concerns about his RNM to Julie Welsh to flag
a complaint through POCL. I have explained to the PM
that as there is nothing wrong with his sytem
(software wise) we are unable to help him." "This
may be a training issue with PM. Have noticed he
has logged a lot of calls, and some days more
than one. On one day in particular he logged 4
calls, and most of the others there are 2 to 3
calls logged since the beginning of this month.”
I surveyed the PP population for any final defect cause being assigned “General - User” or
“General - User Knowledge” which resulted in 435 PPs being identified across a variety of
products. Please keep in mind that the SMC was supposed to resolve user issues. These
PPs were promoted to the SSC.
This figure indicates a wave of user issues around September 1999, March 2000, June 2000,
and November 2000 during the national rollout period.
76
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 8.1 Monthly volumes of PPs
77
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
911
9.1.2
9.1.3
Hardware issues were problematic during Na
nal Rollout
Failure of hardware components in a system can frustrate users and impede the utility of
the system.
In the national rollout of the Horizon IT System, there was a discrete period (August 1999)
where hardware issues rose to the level of being a “serious acceptance incident.” According
to ICL Pathway’s Monthly Reports, some issues persisted through October 1999, but appear
to have subsided to acceptable levels by January 2000.
The following tables contain verbatim extracts from the monthly reports (MRs) and PinICLs
and PEAKs (PPs) which I relied on in identifying this theme. I have intentionally not made
any corrections to grammar or spelling. Where I deemed it helpful, I have highlighted certain
sections in bold. The views expressed in these extracts are that of the authors, being
principally ICL Pathway, but in some cases ICL Pathway and POCL.
Table 9.1 Verbatim extracts from Monthly Reports
URN
FUJ00058185
FUJ00058186
FUJ00058187
FUJ00058188
Title
ICL Pathway
Monthly
Report -
August 1999
ICL Pathway
Monthly
Report -
September
1999
ICL Pathway
Monthly
Report -
October
1999
ICL Pathway
Monthly
Report -
November
1999
Date
August-99
September-99
October-99
November-99
Extracted Text
As anticipated last month, the problems experienced by the
live trial outlets with the Epson back office printer ‘hanging’
during the production of the weekly cash account became a
serious acceptance incident which is proving extremely
difficult to resolve.
Another problem, which occurred at the same time, was
disk time-outs being reported on the Wigan Correspondence
Server. This is suspected as a hardware fault and is being
investigated as such.
298: (Tony H and Dave H) The four week observation
period will start on 21/10. (CCN555 has been raised to
make the observation Cash Account Week integral.) All fixes
are available and a tracking document to record progress
set up. On the cut off date of 1/10 the test-sample was
established as 782 eligible rolled-out outlets representing
1777 eligible counters. The target is a figure of merit of four
units per counter per year, a unit being an authorised
reboot or various numbers of workaround. The CAP 28
figure result was around five units on a very good trend. For
CAP29 the result rose to around seven units because of
376-type issues (see above), new offices not being brought
up to current software revision levels immediately before
first use and some offices not yet equipped with fixes for
printer incidents.
I was very pleased that we were able to meet the
demanding reboot levels categorised under acceptance
incident 298. This was achieved in spite of the serious
Energis switch problem which generated a large number of
NT blue screen incidents. The team is now focusing on one
outstanding counter printer issue, which if resolved, will
ensure that the level of reboots is well within the long-term
objectives.
78
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUJ00058188
FUJ00058189
FUJ00058189
FUJO0058190
Title
ICL Pathway
Monthly
Report -
November
1999
ICL Pathway
Monthly
Report -
January
2000
ICL Pathway
Monthly
Report -
January
2000
ICL Pathway
Monthly
Report -
February
2000
Date
November-99
January-00
January-00
February-00
Extracted Text
Eicon believe that the current connection issues will be
resolved by upgrading the drivers within the Counters. This
is currently under test and distribution to a sample of 200
Outlets is planned over the next few days.
There have been a number of incidents requiring code fixes
to the EPOSS reconciliation reports facility introduced into
the network in late December, and a few faults identified in
the counter applications themselves, otherwise the 4th line
support effort for the live system is in line with our resource
planning expectations. The recent fix to the counter printer
has reduced the number of reboots occurring in the outlets
to a level far exceeding the target agreed with Post Office.
AI 298 authorised reboot counts were down to half the limit
in January and further declined following changes for the
counter printer faults, which had represented about 60% of
the problem. CS is replacing the current manual reporting
process with automated weekly reports covering the whole
estate now that roll out has restarted.
Data Centre performance has been very good with the only
problems reported being hardware failure on the
Correspondence Servers at Wigan which were quickly
repaired. There is still an outstanding issue with the Audit
Servers, which appear not to be built in accordance with the
Technical Description. OSD are investigating.
79
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
9.1.4
It should be noted that in May 2000, there were still hardware issues being raised in PinICLs.
Table 9.2 Verbatim extracts from PPs
URN
Fuj00029091
FUJ00030674
FUJ00030930
FUJ00031124
FUJ00034604
FUuJ00042700
FUJ00046317
Ticket
Source
PinICL
PinICL
PinICL
PinICL
PinICL
PinICL
PinICL
Date
11/08/1999
18/10/1999
22/10/1999
27/10/1999
24/12/1999
17/05/2000
22/05/2000
Extracted Text
"PM has said that this is the 3rd week that his system has hung at
this point, he said that if he usually leaves it for 15mins and it
continues working - so I advised him to wait another 10 mins and
to call back if it hasn't worked by then. He isn't very happy that
this happens week after week and would like it investigated
please." "Have spoken to PM. He is concerned that this has
happened for the last 3 weeks. He does his balancing and gets to
the part of printing the cash account off and it does not print. He
has been leaving the system for a long time and it still does not
print. He eventually gets fed up and does a soft reboot and then
everything is fine. He states that an engineer has been to site to
check his printer and the engineer says that nothing is wrong with
the printer so it is not a hardware fault.” "PM knows how to get
out of the problem but is fed up with it and would like to know
when the problem is going to be sorted." "I have spoken to the PM
who has advised that four the last four weeks he is having
problems printing his Cash Account. What happens is that for
yesterday at approx 2-20 to 2-40pm when he has pressed CA and
the trial balance printing is iniated it takes some 10-12 mins to
print. This he finds unacceptable (as he has waited up to 6 o'clock
and he then re-boots the PC and follows the CA process again and
this time the whole 18 pages of the CA final including the trial is
printed in some 10-15 mins! This he has done for the last four
weeks.”
"printer offline" message when using APS cards” "barbara
suggested that i chase smc regarding engineer, going out to site."
" In the light of this conversation, I am returning this call to SMC
as hardware issue.” "Defect cause updated to 38:General -
Hardware fault"
"[ have spoken to the PM who has advised that they are
experiencing a hardware issue with counter 3. ie they cannot
seem to shut off the power to the counter." "SMC could you
arrange for an engineer to visit this site 306511 to check counter
3. I believe the PM has not contacted HSH still regarding this
hardware issue against my advice."
"the horizon system that she has is extremely faulty, touch screen
has things appearing on it for no apparent reason. also pm has
reported that scanner does not scan. pm also cannot swipe any
cards. pm has tried to enter them manually, but system will not
accept details manually.” "Defect cause updated to 38:General -
Hardware Fault"
“Phantom transactions appearing on the stack...She says that this,
is occurring on all counters. She also mentioned the system going
to a different screen when she was in the middle of 5 P & A
transaction. This type of problem is suggesting keystrokes being
generated by the hardware.”
"Critical TEC messages received for H38442200109 - An
unexpected error occurred while attempting to insert a message"
"I beleive that this counter is suffering a hardware issue" "Defect
cause updated to 38:General - Hardware Fault"
"The hardware reliability of the MCPERSON touch screens needs to
be investigated. Evidence of usage during the testing phases has
shown them to deteriorate with vertical lines obscuring the
display. Of the 10 flat screens on the BTC6 test rig 1 is showing
80
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN Ticket
Source
Date
Extracted Text
FUJ00045452PinICL
23/05/2000
signs of deterioration after 6 months usage. A similar percentage
can be seen on other rigs. In the live this would represent
replacing 10% of the screens every 6 months." "Has a similar tend
been shown in live? Do we have a reliability problem with these
screens?" "The McPerson FPD unit was one of two possible units
that we were considering fo rthe roll out. As it turned out, we only
ever bought a few hundred of these, due to comercial issues and
the inability to resolve certain design changes / requests.
McPerson are not providing any support to us, and therefore whilst
we would normally be keen to determine whether a product is
failing and what levels of failures, this information will go no
where and we have "got what we have got" in this case. Thanks
for the feedback though - is the CTX better in your experience?”
"reports at office 070116 that the total no of tps transactions
totals 169, while the counter totals 1268.i can not account for this
difference on any other reconciliation report. please investigate"
"The fact that there were hardware problems with this counter
position around the time of the ‘rogue’ message being inserted
indicates that this is probably the cause of the out of sequence
message." "Closing call as hardware fault." "Defect cause updated
to 38:General - Hardware Fault"
9.1.5 I surveyed the PPs for the Product at Fault being either “Desktop” or “Hardware”. 1,281 PPs
were identified. There were noticeable maximums in 1996 and then throughout the national
rollout period.
Figure 9.1 Monthly volumes of PPs
9.1.6 I surveyed the PP population for any Product Groups listed in the following figure’s legend.
For the General/Other/Misc legend entry, only PPs where the Product at Fault value was
either “Hardware”, “ISDN”, or “ISDN Adapter/Driver” were included. Similar to the prior
figure, maximums existed in 1996 and through the national rollout period.
81
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 9.2 Monthly volumes of PPs
9.1.7 Of the 332 KELs reviewed 35 of these were coded as Responsive and the Nature of the KEL
was recorded as being hardware-related (i.e., the known issue related to previously
identified hardware issues).
82
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
10. Many Post Office branches were
connected from the
central system during national rollout.
10.1.1
10.1.2
10.1.3
10.1.4
10.1.5
10.1.6
The ambition of the LHITS was to allow branches to communicate their information to a
central system (the LHITS Campuses, as described in section 4.5). It also allowed for
software and reference data updates to be distributed from the campuses to the branches.
To accomplish this design feature, a telecommunications system was incorporated into
LHITS. The telecommunications design depended on ISDN lines (or in some cases satellite
links) being installed at each branch with BT and Energis providing the backbone
infrastructure to utilize this hardware. It also relied on each branch's equipment to be
available for polling (a term that is used when a central system tries to communicate with a
remote system, in this case the hardware installed at the branch).
The Monthly Reports indicate throughout 1998 and 1999 that ICL Pathway was concerned
with their ability to effectuate this design feature: they were concerned with BT's coverage
of the UK as well as other technical issues related to their standards.
During the national rollout these problems were realized. Hardware, network availability,
and user issues combined to create a situation where ICL Pathway was occupied with a
higher-than-expected amount of non-polling branches. This was problematic because the
LHITS relied on this telecommunication design aspect to not only to collate and centralise
information on all of the activity of the branches, but to also allow for efficient updates of
software to the branches.
Additionally, ICL Pathway was compelled to raise and resolve an issue for any branch whose
non-polled status was 24 hours in duration. It is important to understand that this situation
included branches who simply powered down their equipment for a day.
The following tables contain verbatim extracts from the monthly reports (MRs) and PinICLs
and PEAKs (PPs) which I relied on in identifying this theme. I have intentionally not made
any corrections to grammar or spelling. Where I deemed it helpful, I have highlighted certain
sections in bold. The views expressed in these extracts are that of the authors, being
principally ICL Pathway, but in some cases ICL Pathway and POCL.
Table 10.1 Verbatim extracts from Monthly Reports
URN
FUJO0058161
Title Date Extracted Text
Pathway Monthly March-97 We resolved the migration issue which threatened to
Report - March increase our implementation costs but have still to find an
1997 acceptable solution to the limited counter space issue. I
am concerned that BT are failing to implement ISDN
across the UK in the expected timescales. Observers
believe that they could be as much as 2 years behind
schedule which could obviously have serious implications
on our roll out plan.
83
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUJ00058170
FUJO0058186
FUJO0058186
FUJ00058186
FUJ00058187
FUJ00058187
Fujo0058188
Title
Pathway Monthly
Report - March
1998
ICL Pathway
Monthly Report -
September 1999
ICL Pathway
Monthly Report -
September 1999
ICL Pathway
Monthly Report -
September 1999
ICL Pathway
Monthly Report -
October 1999
ICL Pathway
Monthly Report -
October 1999
ICL Pathway
Monthly Report -
November 1999
Date
March-98
September-
99
September-
99
September-
99
October-99
October-99
November-
99
Extracted Text
The inability of the current counter configuration to work
with BT's new ISDN standard is of considerable concern.
We are considering proposals for resolving this but in the
mean-time our ability to deliver operational business
change is significantly hampered.
The expansion of the live estate has meant that the
number of outlets not returning transaction details to TP,
due to ISDN problems or simply that the terminal is
powered down, has increased. This is becoming a job in
itself to track and resolve. We are obliged under the
rectification plan for AI376 to raise an incident on each
office that hasn't polled. This is time consuming and
probably pointless for those offices only down for 24
hours. Richard Brunskill is due to talk to the customer
(with the Requirements team) to try and find a more
efficient way of tackling this problem.
The number of non-polled Post Offices has been increasing
in line with rollout. The task in managing these is
increasing and we need to improve the process and root
cause analysis before we have a significant increase in the
numbers.
A very busy month in the incident management and MIS
areas. We have been dealing with a large number of 'non
polled’ Post Offices as the live estate has rolled out,
causing a bottleneck of incidents which we are only now
beginning to clear. As more offices have become live, the
demands on the MIS team have increased, as there is a
need to monitor the performance of the system for the
acceptance rectification plan. There has also been a
significant amount of time spent re-working SLAs to
portray the accurate values for the SRB.
Energis/BT have just informed us that there are many
more Post Offices which cannot be connected to the ISDN
network despite all their previous work over the last three
years. We are now pressing hard to get a clearer
understanding of the issue and to work out what a
resolution plan would need to be.
A fault has been identified in the EICON card which
supports the ISDN communications protocol. This is
seriously impacting our ability to distribute software
updates to the counters in an efficient manner. It is also
responsible for generating many unnecessary and long
calls resulting in additional network charges. We have
been in contact with the European Service Organisation
and EICON support in Montreal to help resolve the
problem quickly.
As part of the AI376 rectification plan, MSU presented to
POCL TIP the incident management process for business
critical incidents raised by POCL or via the newly
developed EPOSS exception report set. Initial comments
received from TIP were favourable and they applauded
the tighter management controls that ICL Pathway is
introducing.
84
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUJ00058188
POL00029222
FUJO0058191
FUJ00058191
FUJ00058192
FUJ00058192
FUJ00078051
FUJ00058196
Title
ICL Pathway
Monthly Report -
November 1999
Monthly Incident
Review - March
2000
ICL Pathway
Monthly Report -
May 2000
ICL Pathway
Monthly Report -
May 2000
ICL Pathway
Monthly Report -
June 2000
ICL Pathway
Monthly Report -
June 2000
ICL Pathway
Monthly Report -
July 2000
ICL Pathway
Monthly Report-
November 2000
Date
November-
99
March-00
May-00
May-00
June-00
June-00
July-00
November-
00
Extracted Text
Non-polled offices are still creating a large number of
incidents. MSU are identifying where there is a specific
system problem preventing the Outlet from polling.
However there is still the problem where MSU suspect that
Outlets are turning the Counter equipment off evident
from Mondays reports which contain 3 to 4 times the
number of non-polled Outlets than other days within the
week.
The most numerous incidents were for the Non-polled
incident class, accounting for 245 incidents received or
56.5%. This was followed by "receipts and payments"
(migration) comprising 140 of all incidents received, or
32.3%. Please refer to table and reports 3.1 to 3.3 for
further detail.
The War Room set up to address the issue of non-polled
offices has been successful in removing all FADs that were
not polled for over 10 days and the effort is now geared to
get those over 5 days removed.
Following the non-polling exercise conducted with the
involvement of key areas within Pathway, MSU are now
using the revised processes to initiate the resolution of
problems causing Outlets to fail to poll. Early indications
show that the number of non-polled Outlets appearing on
the non-polling report in excess of 5 days is now reducing.
We are still experiencing a number of non-polled outlets in
the live estate. This impacts our file delivery service level
agreements because the transactions cannot be harvested
from these outlets in the required timeframe. The current
t-rust is to ensure that we have resolve all the system
issues and to improve the quality of the various reporting
facilities available to customer services.
Improvements in management processes around the
identification and resolution of non-polled offices have
significantly reduced the amount of offices appearing on
the report. Developments are being identified which
should give earlier warning of Outlets that have lost
communication with the Data Centres. There is an issue
with regard to 100% achievement of 'day D' data
deliveries, which needs to be resolved with POCL.
We are still experiencing a number of non-polled outlets in
the live estate. This impacts our file delivery service level
agreements because the transactions cannot be harvested
from these outlets in the required timeframe. The current
thrust is to ensure that we have resolve all the system
issues and to improve the quality of the various reporting
facilities available to customer services.
The second problem is the increase in the number of post-
offices remaining on the non-polled list. This is mainly due
to problems in SMC staffing due to sickness and time
spent on counter migration activity. A number of
mitigation measures have been put in place.
85
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
Title Date
Extracted Text
POLO0029221 Monthly Incident November- The most frequently occurring incidents in November were
Review - 00
November 2000
both types of Receipts and Payments Incidents (Migration
and Post Migration), with 31 incidents per category. The
Migration incidents have remained at the same level,
whereby the Post Migration occurrences have increased.
This was followed by 17 Transactions Polled by TIP but not
by HAPS, these were due to delayed transactions as
reported on APSS 2133c. These transactions are added
back into normal processing.
FUJ00058197 ICL Pathway December- _Non-polled offices and Day D: John Pope continues his
Monthly Report - 00
December 2000
work with Customer Service and Development to identify
the key issues affecting these areas and helping to
identify solutions such that we can achieve our contractual
SL's.
FUJO0058197 _ICL Pathway December- The non-polled Outlets continue to be high for this period.
DATE
1 DAY
2-3
DAYS
49
DAYS
10-19
DAYS
20+
DAYS
DATE
1 DAY
2-3
Days
4-9
Days
10-19
DAYS
20+
DAYS
Monthly Report - 00
December 2000
A new resource has now been drafted in to aid with the
Non-Polled Outlets that will input further into the
investigation and resolution of these offices.
Table 10.2 Non-Polled Offices - November 2000%7
01- 02- 03- 06-
NOV NOV NOV NOV
07- o0s- o9- 10- 13- 14- 15-
NOV NOV NOV NOV NOV NOV NOV
86 70 77 248 118 95 94 119 383 164 195
83 61 40 130 47 61 64 57 112 45 52
28 22 43 69 67 61 49 53 85 66 53
0 1 1 6 4 5 8 at 13 12 8
0 0 0 0 0 0 [-} 0 0 0 1
16- bE 20- 21-
NOV NOV NOV NOV
115 137 481 284
a 23- 24- 27- 28- 29- 30-
NOV NOV NOV NOV NOV NOV NOV
429 150 141 921 234 151 261
70 65 160 75 76 86 83 153 80 7 92
39 35 74 66 61 51 70 150 128 118 104
8 4 3 2 4 5 3 9 11 8 6
i ° te} i} 0 i) i} ty} 0 1 4.
10.1.7. A review of the PPs shows that issues were raised and resolved for branches whose
equipment had been offline.
Monthly Incident Review - November 2000
(P0L00029221).
One example in the chart indicates that a non-polling issue
86
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
was related to the LHITS. Other non-polling issues were due to factors outside of ICL
Pathway’s control, such as power outages and BT-related circumstances.
Table 10.3 Verbatim extracts from PPs
URN
FUJ00032275
FUJ00031206
FUJ00031675
FUJ00032062
FUJ00035068
Fuj00046403
Fuj00062520
10.1.8
Ticket Date Extracted Text
Source
PinICL 04/10/1999 "FAD 223329 - has appeared on a non polled report, it is one day
late, can it be pinged and why has it not been polled." "I will
check that the pm has rebooted the counters. all four counters
have pinged and the OD is alive on all four counters." "Comms
issue - now resolved. PO has polled, request call closure."
PinICL 25/10/1999 "Office has appeared on the non poll report it is 1 day late" "BT
see line as ‘out of order’ and will investigate" "This was a comms
issue that has now been resolved, PO has polled OK."
PinICL 27/10/1999 “office 265511 has appeared on non polled report, this is one
day late, this is the first day, can it be pinged and why has it not
been polled." "have spoken to PM who advised that there is a PO
sign outside and also that BT were working up the pole outside
yesterday and cut off his other lines as well as the pub next
door" "This fault is still in hand with BT it looks like a problem
between the customer site and the BT exchange. BT are treating
this as their fault and still have this in hand." "This Office has not
polled for 2 days”
PinICL 10/11/1999 "fad 172401 has appeared on the non polled report on 2
separate occasions between the 3rd and 10th of November."
"The non-polling can be attributed to power cuts and comms
probs over the last week. PO has now polled OK - request call
closure.”
PinICL 04/01/2000 "This office is still not polling and hasn't polled for 11 days -
please resolve asap." "Missing objects relating to EPOSSRec were
inserted today by P. Carroll. The PO should disappear from the
non-polling report tomorrow." "The FAD is still on the non-polling
report but the number of days has decreased to 4. The
underlying data when looked at this morning shows the PO to no
longer be a non-polling PO. This means that the non-polling
report is being run too early in the harvesting schedule and is
thus not producing reliable figures." "FAD 181611 still not
polling” "This site is no longer on today's non polling report for
5/1"
PinICL 27/06/2000 “FAD 358136 on non polling report for 3 days.” "BT have advised
jumpers and modules have been reterminated at site." “Comms
to outlet now re-established. However, have checked POStatus
object on correspondence server and this has not yet been
updated with missing EOD's.” "POStatus object now updated at
the correspondence server. FAD is not on today’s non-polled
report.”
PEAK 27/06/2000 "FAD 132859 on non polling report for 3 days.” "Office had not
been polling due to a comms issue - destination out or order.
This has now been fixed and the office is no longer on the non
polling report."
I surveyed the PP population for entries where the Product at Fault contained “ISDN” or
“VPN” within their values. ISDN and VPN are both related to connectivity. This query
resulted in the 4,733 entries shown in the figure below, with their specific Product at Fault
values shown in the legend. This problem manifested during the national rollout period.
87
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 10.1 Monthly volumes of PPs
88
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
11.
nan
I concerns were considered by ICL Pathway
throughout the time period reviewed
11.1.1
11.1.2
11.1.3
11.1.4
11.1.5
11.1.6
ICL Pathway is a profit seeking entity. I assume that ICL Pathway was motivated to deliver
the system and make a profit.
The ICL Pathway Monthly Reports had two sections (‘Commercial and Financial Report’;
‘Business Development Report’) that assessed how the Horizon project was tracking against
internal financial goals and how the Horizon project could be used for other commercial
pursuits.
The financial success of the Horizon IT System relied on ICL Pathway orchestrating many
different constituencies: sponsors, suppliers, and their many internal groups. Benchmarks
relating to acceptance, the timing of rollout coverage, and adherence to SLA requirements
were topics of discussion as they were directly connected to revenue, cost, and profit
realisation. The balancing act between operational and technical activities and their financial
ramifications was highlighted in the April 1998 report: “Should the pressures mount, the
temptation to hold to NR2 dates at all costs is immense. If we were to (purely theoretically)
compromise NR2 quality in order to hold the timescales, we would almost certainly be worse
off in the long run.”
Acceptance was achieved on 24 September 1999, triggering the first invoice to be issued by
ICL Pathway. The rollout coverage incentive for 1,800 outlets was achieved on 5 November
1999. The first payment (£105 million) was received in early December 1999.
The items I have included in this table illustrate some financial wins and some financial
losses from ICL Pathway's perspective. Financial concerns were weighed against the
resource allocations to deliver the Horizon IT System. It is my opinion that the financial
aspects of delivering the Horizon IT System affected the decision-making process.
The following table contains verbatim extracts from the monthly reports (MRs) which I relied
on in identifying this theme. I have intentionally not made any corrections to grammar or
spelling. Where I deemed it helpful, I have highlighted certain sections in bold. The views
expressed in these extracts are that of the authors, being principally ICL Pathway, but in
some cases ICL Pathway and POCL.
89
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
EXPG0000001
EXPG0000001
Expert Witness Report of Charles Cipione, dated 14 September 2022
Table 11.1 Verbatim extracts from Monthly Reports
URN
FUJ00058160
Title Date
Pathway Monthly
Report - February
1997
February-97
FUJO0058161 Pathway Monthly March-97
Report - March
1997
FUJO0058166 Pathway Monthly
Report - December
1997
December-97
FUJO0058166 = Pathway Monthly December-97
Report - December
1997
FUJO0058169 Pathway Monthly
Report - February
1998
February-98
FUJO0058173 — Pathway Monthly
Report - May 1998
May-98
Extracted Text
We have issued the next full risk report on the
programmed and this now includes not just the high
level risks but the comprehensive analysis of all known
risks. This at a high level shows that we start the year
with some £80m worth of risk and that if we pursue our
mitigation actions as currently defined, we can bring
this down to £27m by the end of this year. There is the
management view that there is scope within this to be
more aggressive in reducing risks and this will be driven
through during Q2, Q3 this year.
The Change Control process is now underway. Suppliers
have been given planning data from which to assess the
impacts of the delays. The Suppliers Forum has been
notified of the solid state of the Replan and committed
itself to facilitating the changes. However, the
indications are that we will come under pressure to
recompense suppliers for moneys lost today, never
mind the claw back in eight years time. As predicted
last month, this is likely to be a tough round of
negotiations.
Holding our suppliers is becoming increasingly costly
and fraught for them and our own people who have to
deal with them. We must strike a balance between
saving money and keeping core programme
capability intact. On the one hand, we must resist
paying people simply to stand by and filling warehouses
with equipment we do not need for another year. On
the other, we cannot afford to throw away supplier
goodwill or make life impossibly difficult for our own
staff.
We declared our contractual position formally on 19th
December. In short, we have said that to compensate
us for the programme delays we require either:
© A 30% price increase, or
© A 5% price increase plus a 5 year extension of term.
We are now in major dispute with POCL on the
condition of their physical estate. This has been building
up for over a year and we now have facts and figures to
substantiate the argument that the total cost for
putting their estate into a fit purpose for automation is
on the wrong side of £40m. They appear to have
provided no budget for this, yet their contribution needs
to be close on to £20m.
More supplier tensions are probable over the summer.
Cumulative delays are testing their patience and they
are increasingly looking for near term cash returns.
90
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
Title
Date
EXPG0000001
EXPG0000001
Extracted Text
FUJ00058171
FUJ00058173
FUJO0058158
FUJO0058158
FUJ00058198
FUJ00058183
FUJ00058184
FUJ00058186
Pathway Monthly
Report - April 1998
Pathway Monthly
Report - May 1998
Pathway Monthly
Report - August
1998
Pathway Monthly
Report - August
1998
Pathway Monthly
Report- December
1998
ICL Pathway
Monthly Report -
June 1999
ICL Pathway
Monthly Report -
July 1999
ICL Pathway
Monthly Report -
September 1999
April-98
May-98
August-98
August-98
December-98
June-99
July-99
September-99
NR2 versus NR2+: Should the pressures mount, the
temptation to hold to NR2 dates at all costs is.
immense. If we were to (purely theoretically)
compromise NR2 quality in order to hold the
timescales, we would almost certainly be worse
off in the long run. We are only allowed 11 high or
medium severity Acceptance faults in total: if we fall
foul of Acceptance, we will have to do remedial work
and go round the loop all over again: the delay would
be greater than if we had got it right first time. Unless
NR2 is truly scalable it will need to be replaced very
quickly. If we push too much work out of NR2 into 2+,
the time gap between 2 and 2+ will inevitably increase.
Having NR2 available early but with a dependency on a
rapid follow-on NR2+ does us no good whatever. There
is no point starting rollout unless we know we can keep
it going: start/stop/start would kill us.
The design and build cost profile of the business case
has deteriorated further as a consequence of the cost
increases, placing further demands on additional
funding. Submissions are being prepared to secure
these. The Treasury review will play in.
We continue to underspend forecast because of
recruitment lag. In August, the underspend was £1m
out of £10m. That saves us money in the short term
but means we are not consuming the work at the
required rate.
The cost to date of putting back the programme in
terms of subcontractor settlements now runs into many
£m’'s. We have budgeted for more to come over the
next 18 months.
We have also included within our business plan
contingency funds to cover the critical known risks. The
single largest of these is any risk or delay to the
beginning of National Roll-out and possible problems in
maintaining the beat rate of implementation. These
contingency plans are an essential component of Risk
Management which we must have in a programme
which has elements of risk between medium and in
some cases high severity.
We are determined to meet the cash payment points
which follow Acceptance (£68m) and successful Roll-out
to the first 1,800 Post Offices (£90m) which are vital
payment points in 1999 both for ICL/Fujitsu funding
and of course for the credibility of the new programme
moving forward. All staff are focused on the criticality of
meeting these milestones.
We are still determined to meet the cash payment
points which follow Acceptance (£68m) and successful
Roll-out to the first 1,800 Post Offices (£90m) which
are vital payment points in 1999 both for ICL/Fujitsu
funding and of course for the credibility of the new
programme moving forward. Alll staff are focused on the
criticality of Meeting these milestones
Acceptance was achieved on 24 September and the
resultant invoice for £68m delivered on 27 September
for payment within 30 days.
91
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUJ00058187
FUJ00058187
Fuj00058189
FUJ00058191
FuJ00058195
Title Date
ICL Pathway October-99
Monthly Report -
October 1999
ICL Pathway October-99
Monthly Report -
October 1999
ICL Pathway January-00
Monthly Report -
January 2000
ICL Pathway May-00
Monthly Report -
May 2000
ICL Pathway October-00
Monthly Report -
October 2000
EXPG0000001
EXPG0000001
Extracted Text
The ‘upside! roll out target of 1800 post offices was
achieved on Sth November, enabling us to invoice the
higher figure of £90m. rather than the alternative Of
£80m for 1600 post offices. An invoice for the
£105,162,500 including VAT was couriered to POCL on
Monday 8th November, and POCL's confirmation of
receipt was received on 9th November. Payment is due
on Sth December, which is another Sunday.
As doubtless reported elsewhere by my colleagues,
there has been intense activity on the post Acceptance/
pre January roll out decision front, and this will intensify
further in the run up to 24th November and beyond.
Reference data is the big issue. Some resolutions must
be found before roll out can safely restart on 24th
January. As a measure of the exposure, we face a claim
from POCL (which we are disputing on the grounds that
they authorised it) of some £300k in respect of just one
reference data error.
£105m payment for the full first Roll-out milestone was
received on time in early December.
Weekly service performance is a key issue and recent
problems with Help Desk service have significantly
dented PO confidence. March and April were disastrous
months on OSD service levels, driven by major
resource issues (staffing levels) on the Horizon System
Help Desk. Nearly all Of the SLA's have been missed
and significant penalties incurred. This is an own goal
and should have been prevented. As reported last
month it is on Red Alert and OSD have reacted
decisively and professionally to implement corrective
action. Their management has been changed and over
40 new help desk staff recruited along with a plan to
recruit at a pace to handle the weekly increase in Post
Offices and to cover for attrition. This has driven a
dramatic improvement and this week we are now back
on target with 7 of the 10 key SLA’s. The sensitivity of
this situation cannot be overstated. It is highly visible
and has brought firm reaction from PO Directors. It will
take week on week, month on month good performance
to recover our position.
Weekly service performance remains a key issue and
although we are back on track and demonstrating
consistent performance we are missing some of the
very challenging SLA's: As expected PO have now
placed us in formal Breach of Contract (they can do this
if we miss any three quarters in 24 months) although
they currently appear to be genuinely seeking
contractual compliance rather than financial
recompense. A meeting is fixed for mid November with
the customer to try and finalise a way ahead on this.
My concern is that we have breach and termination
hanging over us on an ongoing basis. Also that we
establish a methodology that avoids the withholding of
our second £60M retention by PO from July 2001.
92
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN Title Date Extracted Text
FUJO0058196 —_ICL Pathway November-00 — CSR+ counter migration has been largely completed.
Monthly Report- Acceptance of CSR+ has triggered payment of the first
November 2000 £60M retention starting in January 2001 at £1.25M per
month. The quality of CSR+ appears robust (as
evidenced by help desk calls). Work is underway to
crystallise and achieve the requirements for the second
£60M retention due in Q2 next year.
11.1.7 Based on my understanding of the content of the Ps, I do not believe any of their content
would be relevant to ICL Pathway’s financial concerns, and therefore I did not undertake
searches of the documents for this theme.
93
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
12.
The tenuous relationship between ICL Pathway, their
sponsors (BA and POCL) and suppliers were often topics of
concern for ICL Pathway's management team
12.1.1
12.1.2
123.3
12.1.4
12.1.5
12.1.6
12.1.7
12.1.8
12.1.9
12.1.10
12141
12.1.12
ICL Pathway dealt with many different groups during the design, development, and
deployment of the Horizon IT System. Throughout this period, I noted that ICL Pathway
had recurring communication and expectation management concerns with these groups.
Early in the process (1997) ICL Pathway’s interactions focused mainly on sponsors. It is
clear from the March 1997 entry that ICL Pathway believed much of the responsibility for
preparing the branches for automation lies with the Post Office network itself. This indicates
an initial mismatch in the expectations of the Horizon project. ICL Pathway was not
comfortable with the Post Office network's position on its readiness.
The timeline slippage also concerned ICL Pathways’ suppliers, who had invested in
equipment and were now told they need to keep this inventory longer than expected.
Continuing the timeline slippage motif, August 1997 saw the three partners (ICL Pathway,
POCL and the Benefits Agency ("BA") experiencing damage to their business cases. At this
point ICL Pathway speculated that POCL and BA might be considering alternatives.
In February 1998, ICL Pathway’s difference of opinion with POCL about counter space
problems rose to the level of sending “legal letters.”
In May 1998, ICL Pathway predicted more supplier issues over the summer due to
cumulative delays in the timeline.
In December of 1998, ICL Pathway and POCL targeted August 1999 for national rollout. In
January 1999, BA “unilaterally” pushed out their timeline by six months. ICL Pathway and
POCL represented against this decision, “some of it quite legal in nature”.
In April 1999, BA (DSS) removed themselves from the Pathway project.
In May 1999, ICL Pathway found itself needing to “rebuild customer relations following the
traumatic arrangements that brought the new contract into force”.
In June 1999, ICL Pathway indicates that POCL still harbour negative feelings toward them,
believing that POCL blames ICL Pathway for the way POCL has been treated publicly.
In October 1999, POCL accused ICL Pathway of resorting to undue escalation related to
addressing reference data issues.
These relationships needed to maintain a healthy hygiene of clear communication and
orchestrated manoeuvring as issues presented themselves. It is my opinion that at several
points in time, the parties’ individual goals and expectations were at odds with each other.
This diversion of focus could not have benefited the implementation of the Horizon IT
System.
94
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
12.1.13
The following table contains verbatim extracts from the monthly reports (MRs) which I relied
on in identifying this theme. I have intentionally not made any corrections to grammar or
spelling. Where I deemed it helpful, I have highlighted certain sections in bold. The views
expressed in these extracts are that of the authors, being principally ICL Pathway, but in
some cases ICL Pathway and POCL.
Table 12.1 Verbatim extracts from Monthly Reports
URN
FUJO0058161
FUJ00058162
FUJO0058162
FUJ00058163
FUJ00058163
FUJ00058163
FUJ00058169
Title
Pathway
Monthly
Report -
March 1997
Pathway
Monthly
Report -
June 1997
Pathway
Monthly
Report -
June 1997
Pathway
Monthly
Report -
August 1997
Pathway
Monthly
Report -
August 1997
Pathway
Monthly
Report -
August 1997
Pathway
Monthly
Report -
February
1998
Date
March-97
June-97
June-97
August-97
August-97
August-97
February-98
Extracted Text
As we survey the first 200 offices the extent of this problem
is becoming clearer and worse. We have formally lodged with
PDA that the Post Office network is not fit for the purpose of
automation and that this responsibility clearly lies with the
sponsor. Difficult negotiations must be anticipated.
The programme review has been presented to PDA, POCL,
BA, ITSA and SSA [Northern Ireland]. The consistent
response has been one of disappointment and shock that yet
another slippage has come to the surface so soon after two
previous replans.
We have fully briefed our principle suppliers on the current
position and not surprisingly this causes many of them
serious problems particularly where they have invested in
equipment, resources and capability against a plan which has
moved smartly to the right. Different mitigation actions are
called for with different suppliers but this will be a rough task
to get straightened out before we can move forward with
confidence.
This has been another difficult month for Pathway, BA and
POCL. There have been further programme delays and
difficulties with Release 1c and the potential release date for
Release 2. This has caused further damage to the business
cases of all parties and within the Sponsors there is a body
of people who are pressing and looking for a way out the
contract.
We know that the BA and POCL business cases have been
badly damaged and that there are elements within both
organisations which are now looking for out or at least are
thinking about alternatives. BA will maintain that they have
not slipped since the last replan in April (CCN105):
essentially true as far as we can tell.
Another very difficult month for the programme. There is no
disguising the perception that we are in the dock for the
latest slippage and the fact that the damage to business
cases all round is immense. On the plus side, our new
openness and realism about the state of the programme has
been met with a positive response from the other side and a
preparedness to work with us to find a way through.
The counter space problem has been better defined and is
more serious than we had previously thought. We have gone
on the attack with legal letters, which make it clear that,
beyond a certain point, we consider this to be a matter for
POCL to face up to (and pay for). The current prognosis for
an early and amicable resolution is not good. Meanwhile, we
need to be even more creative and determined to find ways
to address the problem (moneys aside).
95
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUJ00058173
FUJ00058198
FUJO0058168
FUJ00058182
FUJ00058182
FUjJ00058183
FUJ00058187
FUJ00058189
Title
Pathway
Monthly
Report -
May 1998
Pathway
Monthly
Report-
December
1998
Pathway
Monthly
Report -
January
1999
ICL Pathway
Monthly
Report -
May 1999
ICL Pathway
Monthly
Report -
May 1999
ICL Pathway
Monthly
Report -
June 1999
ICL Pathway
Monthly
Report -
October
1999
ICL Pathway
Monthly
Report -
January
2000
Date
May-98
December-98
January-99
May-99
May-99
June-99
October-99
January-00
Extracted Text
More supplier tensions are probable over the summer.
Cumulative delays are testing their patience and they are
increasingly looking for near term cash returns.
The key event is the start of National Roll-out, since this is
the time when we begin to implement the critical
infrastructure in the Post Office Network and on the
completion of this, then the revenue flows to ICL-Pathway
begin in earnest. We and Post Office Counters Ltd are in
agreement that August 1999 is the appropriate date to start
National Roll-out with sufficient contingencies to cover the
likely problems we will encounter. We believe that National
Roll-out can begin in early August, where as POCL view that
it is more likely to be the last week in August. We continue
to press for a more aggressive plan to avoid any
unnecessary delay to this activity.
All plans work towards a National Roll-out date for NR2 in
August 1999. Nevertheless there has, during the last two
weeks, been a major confrontation with the Benefits Agency
who have abruptly and unilaterally moved their multi-benefit
Model Office plans by six months. Following strong
representation from ourselves and POCL some of it quite
legal in nature, DSS have become defensive and attempt to
retreat on the issue. Nevertheless the issue remains untidy,
could lead to press leaks and needs rapid and complete
resolution. This will need to be watched extremely carefully.
We need a major programme to re-build customer relations
following the traumatic arrangements which brought the new
contract into force. We have a substantial way-to go on this
with POCL before this contract is in good order.
The withdrawal of DSS from the Agreements removed seven
of the 25 Acceptance areas. In addition Horizon has indicated
that a further area will not now be pursued. The DSS
withdrawal has also removed the two Acceptance
Specifications that DSS and POCL refused to approve, and
several of the issues that were previously a concern.
Although we are now some six weeks into the new contract
arrangements POCL continue to remain negative and critical
towards the programme and have not yet got over their
bitterness on the way they have been treated within the
public sector, for which unfortunately they continue to hold
us partially to blame. We have to work at this as we make
progress with the commercial, financial and programme
matters in order to find a more positive and long term
relationship.
We were unable to convince POCL of our case over reference
data without resorting to what POCL regarded as undue
escalation. That went down badly and caused a negative
reaction. The right actions now appear to be underway but
we need to establish new conduits for better day to day
communication and * step-by-step escalation’. The actions
have Operations leading, with Programmes in support: this is
a significant change which calls for changes in the ways we
do things internally as well as between ourselves and POCL.
We must keep pushing for a joined up campaign internal to
Post Office to promote Horizon. There are far too many
misconceptions about which vary from out-of-date
technology to a batch system to "it will take 3/4 years to
complete roll-out."
96
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN Title Date
FUJO0058190 ICL Pathway February-00
Monthly
Report -
February
2000
FUJO0058192 ICL Pathway June-00
Monthly
Report -
June 2000
FUJO0058194 ICL Pathway August-00
Monthly
Report -
August 2000
FUJO0058195 ICL Pathway October-00
Monthly
Report -
October
2000
FUJO0058196 ICL Pathway November-00
Monthly
Report-
November
2000
FUJO0058197 ICL Pathway December-00
Monthly
Report -
December
2000
FUJO0058197 ICL Pathway December-00
Monthly
Report -
December
2000
Extracted Text
There were no Network incidents requiring Energis
involvement during the reporting period. However there have
been occasional incidents at Post Offices that are taking too
long to resolve. A letter of complaint has been sent and a
meeting is planned with Energis and BT in order for both to
explain their lack of action.
On a broader front in the Post Office, business is proving
difficult and ICL is not seen as a strategic supplier in Parcel
Force nor Royal Mail. We are still tarnished with the history.
Our PC Supply contract has hit severe difficulties this month
with supply issues from MVC and PO have expressed major
dissatisfaction.
There are some supplier issues also:
* Flat screen quality - Optoma have produced a firmware
mod but site visits are required - there is bound to be a
fight over costs (estimated at circa £1m)
+ Ntl final payment - £80k claim of which we judge
perhaps 25% to be fair
* — KPL- our claim in respect of wasted course places (as
above - their share could amount to £400k)
* Celestica - mobiles cost- £130k disputed cost hike
We have invoked Masons to write to ntl: to knock them back
from their £800k claim for standby charges. We feel strongly
that their claim should be more like £250k. Our action may
provoke some reaction because ntl: are a strategic customer.
We will also be writing to Energis to claim back costs which
have resulted from what we assert were their breaches of
contract with respect to ISDN line installations.
Agreement has been reached with ntl: regarding their £800k
claim. The outcome is circa £350k. This is very close to the
latest (tasked) forecast.
Although we are demonstrating consistent and good quality
operational performance we are missing some of the very
challenging SLA's and as expected PO have placed us in
formal Breach of Contract (they can do this if we miss any
three quarters in 24 months). We are trying to negotiate a
reduced SLA breach trigger for the future that also sweeps
up the training occupancy issue. A proposal is currently being
considered by the Post Office and we appear to be homing in
on a mutually acceptable solution.
Negotiations with Energis for compensation relating to failure
to install ISDN lines as required by the contract remains
ongoing. An offer of £50K has been received and rejected on
the basis that it is a significant under estimation by Energis
of the costs incurred.
12.1.14 Based on my understanding of the content of the PEAKS and PinICLs, I do not believe any
of their content was relevant to ICL Pathway’s relationships with their sponsors and
suppliers.
97
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
13. The per:
tence of reference data mismanagement degraded
the integrity of Horizon
13.1.1
13.1.2
13.1.3
13.1.4
13.1.5
13.1.6
ICL Pathway designed the Horizon IT System to utilise data driven logic. This design
feature’s benefit was to efficiently update the Horizon IT System's functionality without the
need to develop, design, test, and deploy new versions of the software.
POCL was responsible for maintaining portions of the Horizon IT System's Reference Data.
Reference Data maintained pricing information for the different types of stock sold at the
branches. It also contained behind-the-scenes information that was needed by the Horizon
IT System to map accounting transactions properly.
The advantages of data driven logic rely upon its custodianship. If the “data” in the data
driven logic is not timely, accurate, and complete, the system it supports will not operate as
intended.
In early 1997, ICL Pathway identified the need to incorporate POCL’s Reference Data into
the Horizon IT System. By late 1997, ICL Pathway characterised its contractual obligations
regarding Reference Data as “poorly defined” but acknowledged the significance of the issue
as crucial.
The Pathway Monthly Reports clearly represent that ICL Pathway believes the maintenance
of some of the Reference Data is the responsibility of POCL. These same reports also
describe a litany of instances where, according to the ICL Pathway reports, POCL failed in
this responsibility. The resulting Reference Data issues caused errors in the Horizon IT
System.
The following tables contain verbatim extracts from the monthly reports (MRs) and PinICLs
and PEAKs (PPs) which I relied on in identifying this theme. I have intentionally not made
any corrections to grammar or spelling. Where I deemed it helpful, I have highlighted certain
sections in bold. The views expressed in these extracts are that of the authors, being
principally ICL Pathway, but in some cases ICL Pathway and POCL.
Table 13.1 Verbatim extracts from Monthly Reports
URN
FUJ00058160
FUJO0058166
Title Date Extracted Text
Pathway Monthly February-97 There is potential issue concerning the acquisition and
Report - distribution of BPS Reference Data. It needs to join the
February 1997 POCL Reference data stream at some point.
Pathway Monthly December-97 _ Reference data is poorly defined in the contractual
Report - requirements but is crucial for the proper control of
December 1997 changes to outlet/product data. POCL are only now
realising its significance and we must be vigilant if we
are to avoid requirements creep.
98
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUJ00058169
FUJO0058170
FUJ00058170
FUJ00058171
FUJ00058173
FUJ00058174
FUJ00058175
Title
Pathway Monthly
Report -
February 1998
Pathway Monthly
Report - March
1998
Pathway Monthly
Report - March
1998
Pathway Monthly
Report - April
1998
Pathway Monthly
Report - May
1998
Pathway Monthly
Report - June
1998
Pathway Monthly
Report - July
1998
Date
February-98
March-98
March-98
April-98
May-98
June-98
July-98
Extracted Text
The CARs were brought up to date. I sent a letter
withdrawing our approvals for CARs relating to
Reference Data. The subsequent muscular
correspondence has led to a letter to Tony O asking him
to intercede with me on the customer's behalf. This
issue will either resolve itself with deliveries by POCL of
satisfactory Reference Data by the end of March or will
become a severe embarrassment to POCL if they miss
either or both of the quality or the date.
As reported in last months report, the focus of attention
is now the main pass stages of BPS and EPOS system
testing. These activities have been seriously impacted
by a series of problems related to the mapping of
‘reference data to the EPOS counter application. A
number of corrective actions have been implemented
and recovery options are now being evaluated. The
Direct Interface Testing with BA (CAPS & OBCS) and
POCL (RDMC & TIP) has gone well and we are now
poised to start the final stage (i.e. DIT2).
The last month has not been an easy one for the work
on New Release 2 planning and progress. Severe
problems with EPOSS testing within Pathway and linking
through to reference data within POCL have caused a
delay of between three and five weeks to the schedule.
A mitigation plan has been drawn up although this has
high risk and low confidence and discussions are now in
hand with the sponsors to open up the debate on a
better plan to get to LiveTrial in January 1999. This area
will remain extremely difficult for some time.
This has been a busy month to reshape the NR2
planning to achieve the January Live Trial date, which,
due to stress points within EPOSS/Reference Data, has
meant a moving around of internal milestones and the
need for all parties to reconnect on a different schedule
for Model Office testing. The most difficult area here, will
be with BA who saw the contingency owned more by
them than by us and are therefore somewhat reluctant
to co-operate.
For ICL Pathway the stress points in this plan are the
satisfactory completion of the EPOSS system testing
which has been a difficult area with its tight interfaces to
the POCL reference data system. In addition we are
scheduled to enter the direct interface testing of our
systems with POCLs starting on the 16'" June and
currently we have a small number of critical software
fixes to complete.
We have received updated versions of the Reference
Data from POCL but the quality is below that expected.
Meetings have been arranged with POCL in an attempt
to resolve these problems before the start of model
office rehearsals.
The quality and the change control processes associated
with POCL reference data is causing a considerable
amount of rework for the programme. This is delaying
progress on a number of fronts and must be addressed
urgently.
99
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUJ00058175
FUJ00058176
FUJ00058177
FUJO0058186
FUJ00058186
FUJ00058187
FUJ00058187
Title
Pathway Monthly
Report - July
1998
Pathway Monthly
Report -
September 1998
Pathway Monthly
Report - October
1998
ICL Pathway
Monthly Report -
September 1999
ICL Pathway
Monthly Report -
September 1999
ICL Pathway
Monthly Report -
October 1999
ICL Pathway
Monthly Report -
October 1999
Date
July-98
September-
98
October-98
September-
99
September-
99
October-99
October-99
Extracted Text
Reference data continues to be a source of concern even
though we have received several updates from POCL for
the model office and live versions. The end to end
control of this critical data is not adequate but we expect
POCL to make vital organisational changes in August in
an attempt to address this issue.
POCL are finally getting to grips with the end to end
procedures and disciplines required to manage their
Reference Data. The progress over the past few weeks
has been encouraging and if maintained, will ensure we
develop a workable process.
The data transferred to TIP contains transaction details
(ITM's) created by the Pathway solution. These are
inconsistent with the details passed to TIP by POCL
reference data. Discussions with POCL are being held to
determine how this problem can be resolved.
Major operational problems were experienced with Rem
In/Rem Out, Stock-unit. Transfers and scales. On Friday
1 October a number of outlets experienced problems
Rem'ing in and out, transferring stock between stock
units and scales functionality. This was diagnosed as the
office details having a change in the Reference Data
pipeline that caused a previous change to be released
with an end date on the data when the actual change
containing this end date had not been released.
Although the Rem and scales problems were resolved,
the stock transfer amendments were not made until
Monday, which resulted in the number of outlet calls
raised on that day. A resolution plan is in place to
ensure there will be no cash account balance problems
on Wednesday.
The performance of delivering reference data is cause
for concern. The impact of reference data failure has
been evident over the past two weeks where recent
failures have had a major impact at the Post Office
counter. Given an estate of 20,000 Post Office outlets
any reference data failure will cause enormous problems
for the postmaster, the HSH and support teams. The
end-to-end reference data System needs to be fault
tolerant - at present it is not.
Too many reference data errors are being distributed to
the counter. End to end design reviews are being held to
establish what action can be taken swiftly to prevent
these occurring in the future. These are having a major
impact on Acceptance Incident 376. In addition, the
performance of the data distribution process is
inadequate and must be improved before roll-out
commences in late January 2000.
As doubtless reported elsewhere by my colleagues,
there has been intense activity on the post Acceptance/
pre January roll out decision front, and this will intensify
further in the run up to 24th November and beyond.
Reference data is the big issue. Some resolutions must
be found before roll out can safely restart on 24th
January. As a measure of the exposure, we face a claim
from POCL (which we are disputing on the grounds that
they authorised it) of some £300k in respect of just one
reference data error.
100
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUJ00058187
FUJ00058187
FUJ00058187
FUJ00058188
FUJO0058188
FUJ00058189
Title
ICL Pathway
Monthly Report -
October 1999
ICL Pathway
Monthly Report -
October 1999
ICL Pathway
Monthly Report -
October 1999
ICL Pathway
Monthly Report -
November 1999
ICL Pathway
Monthly Report -
November 1999
ICL Pathway
Monthly Report -
January 2000
Date
October-99
October-99
October-99
November-
L I
November-
99
January-00
Extracted Text
We were unable to convince POCL of our case over
reference data without resorting to what POCL regarded
as undue escalation. That went down badly and caused
a negative reaction. The right actions now appear to be
underway but we need to establish new conduits for
better day to day communication and step-by-step
escalation’. The actions have Operations leading, with
Programmes in support: this is a significant change
which calls for changes in the ways we do things
internally as well as between ourselves and POCL.
The monitoring of the three big Acceptance Incidents
(AI298, A1376 and AI408) have all run into difficulties in
varying degrees with the common theme being the
potentially unsafe state of operation of Reference Data
within the end-to-end model. As mentioned already the
end-to-end workshop is the critical process for finding
an acceptable resolution to this complex area.
Too many Reference Data errors are being distributed to
the live estate which has been causing major problems
with reconciliation and cash account production. We are
pressing for a full end-to-end review across Horizon as
well as Pathway such that solutions can be found and
implemented prior to a roll-out restart in January 2000.
The "big three" Acceptance issues have been reduced to
the "big two" with the clearance of 298 (Counter
Stability). Actions have been developed for handling the
issues on EPOSS Reconciliation and Reference Data
sufficient to get us to the decision to restart the rollout
on 24 January. There are new starts on Network
Banking and Euro study.
The third meeting of the Reference Data get-well plan
was held last week. Attendees included Keith Baines,
Tony Qppenheim, Mike Coombs and Martin Riddell. The
meeting was very positive and hopefully increased the
onus on POCL to produce specific action plan to address
data quality issue. Data assumptions document has
been sent to POCL. CS has received POCL Business
Rules document and comments have gone back to
POCL. There is a problem relating to change in passport
price data received from POCL on 29th November. The
price change was linked in to other changes for which
additional work was required and for which we had an
OLA of 4 weeks. The change is required for 16
December. There is a risk that we will be unable to
complete this change for the 16th December. POCL have
been informed and a response is awaited. SIP 16 data
has been delivered to all outlets and a selected number
have been supplied with the code and activated.
Development work on CP2298, the change to
RDDS/RDMC is reported to be progressing to plan.
After only two incidents, one resulting from a mistake by
PO and another from an error in Pathway’s systems, it
was clear that the processes for Reference Data
Management and Authorisation were inadequate. The
key issues of verifying the accuracy of reference data
before its authorisation by PO for propagation to the live
estate, was jointly reviewed and new plan was produced
to enable National Rollout to recommence by 24th,
January. A more robust interface agreement was agreed
on 14th January.
101
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUujJ00058189
FUJ00058190
FUJ00058192
FUuj00058194
FUJ00058196
FUJ00058197
13.1.7
Title Date
ICL Pathway January-00
Monthly Report -
January 2000
ICL Pathway February-00
Monthly Report -
February 2000
ICL Pathway June-00
Monthly Report -
June 2000
ICL Pathway August-00
Monthly Report -
August 2000
ICL Pathway November-
Monthly Report- 00
November 2000
ICL Pathway December-00
Monthly Report -
December 2000
Extracted Text
There have been a number of problems with the
processing of Reference Data in the last two weeks.
These include: A number of files were sent to RDMC that
were thought to be benign to the Horizon system. This
was not the case as it caused some changes to occur in
the Horizon counters that may not have been expected.
The problem was due to POCL failing to supply required
data to a previous OBC in September 1999 to ensure
that a change in AP client names was propagated to all
associated AP tokens. The Reference Data Comparison
tool successfully identified this problem. A number of
files were sent to RDMC that contained a change to the
Cash Account type for both live and non-Live outlets.
Unfortunately a process issue had allowed these
changes to he supplied by POCL as Help Desk changes
only and RDT were also not informed that the files
contained such data. As a consequence RDT were
unaware of the urgency with which the files were
required and had not progressed them for release. The
problem was further complicated in that some of the
files had dependencies on others for which RDT were
awaiting action both from POCL and from ICL Pathway.
RDT were able to progress the files slightly later than
required but we believe that this has not caused any
problems to Live outlets. The underlying cause of the
problem is being addressed and additional processes are
being established within POCL to ensure that this does
not happen in the future. An outlet’s FAD code was
changed in anticipation of-the outlet moving to a new
location and franchise however the actual change was
delayed but this was not reflected by POCL within the
Reference Data. This caused problems for ICL Pathway
data processing. The immediate problem was overcome
by provision of corrective Reference Data however the
underlying issue is still under investigation.
Late delivery of Reference Data or Reference Data
Amendments by POCL causes Pathway problems in
maintaining the scheduled timescales. A process is
being. introduced to trigger an E-Mail to POCL QSG
every time they are. late with these deliveries.
There are still concerns regarding the quality of
Reference Data received from POCL, in particular
regarding the actual volume of errors and the amount of
rework resulting from last minute changes. It is difficult
to identify ownership within POCL to take this forward.
This is being addressed through the HSRF.
There is concern over the performance of the Data
Centre with large volumes of non-core Reference Data
remains, even with 64 Agents at CI_4. Changes may
need to be made to the processing mechanisms. This is
being monitored.
Quality of Reference Data from POCL and time taken to
resolve the problem remains a concern. This has been
escalated within POCL.
Quality of reference data remains an issue. Insufficient
progress is being made by POCL in this area and has
been escalated to BSM.
A review of the PPs supports the contention that Reference Data was a cause of problems in
the Horizon IT System.
102
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Table 13.2 Verbatim extracts from PPs
URN Ticket
Source
FUJ00038157 PinICL
FUJ00039293 PinICL
FUJ00075020 PEAK
FUJ00038613 PinICL
FUJ00039673 PinICL
Date
30/06/1999
01/09/1999
13/10/1999
22/10/1999
23/10/1999
Extracted Text
"Due to a historical problem with reference data, 39 bus ticked
products will not have been appearing on the balance report
causing a misbalance as they will have been transacted at the
counter. A fix for this problem has been developed and is
currently undergoing testing within ICL Pathway." "Product
2533 reference data contains no Primary Mapping
attributes. ICL Pathway has identified that this product (and
approx. 38 other ‘Local Products’) has never had a set of Primary
Mappings applied since the data was delivered from POCL. A
correction to this reference data has been delivered in WP5759,
which has been applied to the live system on 24/9/99."
"Inspection of the message store shows that P&A transactions
undertaken on 28.10.99 did not contain any ‘Primary Mapping’
attributes, indicative that the reference data for the P&A
products was not present when the transactions took place."
"The reason that the value was added to line 5002 was that the
update of the Cash Account mapping reference data for Product
21 which added the new mappings for revaluation failed to
update correctly at the office (known problem already being
addressed at 300+ outlets), the system therefore added the
value to the ‘default! line which is line 5001." "The reference data
problem where mixed mode data was recorded, was resolved
under the reference data change in wp 5760 & 5817, that
prevents movement from revaluation to housekeeping.”
"The error in the system which allowed the P&A transactions in
the above outlets to be recorded while in 'Remittance’ mode has
already been identified and revised Type 'C’ reference data has
been delivered to correct the problem.”
“It is also possible that since the outlet was migrating on 1.10.99
when there was a known problem with the Mode Parameters
reference data and the Pathway Settlement Products Reference
Data that the migration of any Remittances and Transfers from
the ECCO+ system may not have been handled correctly - again,
I would need to see the correspondence server node messages
in order to determine this." "Have not yet been able to devote
any further time to anlysing the source of the final element of
the misbalance, but it is undoubtedly tied up with the reference
data issues which caused the Transfer and Remittance issues
identified in the earlier update."
"The differences reported on the Cash Account originated in CAP
28 when two transfers of cheques (£2252.59 and £2168.89)
were corrupted due to the transfer reference data deletion during
the period 1st to 4th October. As a result, the values for Cash
And Cheques reported on the CAP 28 Cash Account were
incorrect (Cash was reported £4421.48 higher than it should
have been, Cheques £4421.48 lower than it should have been).
This had a knock on effect on TIPs calculation of the CAP 29
Cash Account values since the starting position taken from the
previous Cash Account was already incorrect.”
"The remaining 24 FAD Codes involved in the call were all
affected by the transfer problems in CAP 28. Because transfers
between 1st and 4th October were incorrectly recorded in the
message store (caused by the deletion of the Transfer reference
data) the values of transferred items of STOCK (not Cash) were
incorrectly added onto the declared stock amounts for the office
when the Cash Account was produced for CAP 28. As a result,
TIP used these as the starting stock figures for the CAP 29 Cash
Account and identified the indicated differences during CAP 29."
103
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUJ00032563
FUJ00040565
FUJ00065150
FUJ00066141
Ticket
Source
PinICL
PinICL
PEAK
PEAK
Date
11/11/1999
24/03/2000
28/07/2000
11/08/2000
EXPG0000001
EXPG0000001
Extracted Text
"Inspection of the message store shows that P&A transactions
undertaken on 28.10.99 did not contain any 'Primary Mapping’
attributes, indicative that the reference data for the P&A
products was not present when the transactions took place. The
P&A Product Reference data appears to have been loded onto
Node 38 (the correspondence server) at c. 15:00 on 28.10.99.
SSC/Customer Services need to explain why this reference data
was not available at the outlet from the point of installation." "A
reference data change (from Pathway) is required to prevent
users from navigating between Housekeeping and Revaluation
while there are transactions on the session stack.” "The reason
that the value was added to line 5002 was that the update of the
Cash Account mapping reference data for Product 21 which
added the new mappings for revaluation failed to update
correctly at the office (known problem already being addressed
at 300+ outlets), the system therefore added the value to the
‘default’ line which is line 5001." "The reference data issue
identifed earlier in this call where mixed-mode transactions were
able to be recorded, is being fixed by Peter Morgan with a
release of reference data."
"Under normal circumstances when the user selects the 6p
Stamp button on the Remittance out menu, the system actually
records a sale against Product 21. In this particular case it looks
as though the user may have made use of the PLU number to
directly sell product 609 itself - a PLUImplulses Collection record
for ObjectName 609 was delivered to the outlet on 23rd
February 2000. Since product 609 is not normally sold under its
own product number, transactions against the product number
will fail to report correctly to the Cash Account (there is no CA
Mapping provided to report the decrease in stock to Tables 5b
and 5), causing Receipts to not equal Payments on these
reports." "The solution to this issue would appear to be for POCL
to delete the PLUImpulse records from their reference data for
those products which are not genuine ‘Customer Service’
products."
"I know what caused this problem. It was because reference
data was not sent to the outlets concerning P&A products--The
cash settlement was mapped to the CA, but the corresponding
transaction was not." "This difference in the receipt and Payment
totals was caused by the fact that non-core reference data was
not delivered to this office in time. The reference data was for
OBCS products 177 to 185. As this reference data included
primary mappings for these products these products could not be
mapped to the cash account at stock unit rollover.” "All the
offending transactions took place 21/7/00-- when there was not
reference data at the outlets. The correct reference data was
delivered for business on 22/7/00."
"The original MiECCO unpaid cheque is in mode RISD. This is not
a defined mode for EPOSS and the Cash Account mappings for
product 5 do not have a place defined for RISD and hence the
action is undefined; but ( and I will confirm this later ) probably
use SC serve customer as default which maps it to stock. If it
had been mapped as a ROOP instead then the cash account
would have balanced." "One way of preventing this problem in
the future would be for POCL to provide a sensible RISD mapping
for product 5 mapping it on the receipts side of the cash account
( rather than it being treated as server customer which puts it on
the payments side causing the misbalance )."
104
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
13.1.8 I surveyed the PP population for any Product at Fault value where Reference Data was
indicated. I identified the 1,863 PPs in the chart below, whose legend itemizes the Product
at Fault values. Reference data problems began manifesting in 1998 and were prevalent
during the national rollout period. Interestingly, a Product at Fault value of “POCL Reference
Data” seems to appear in February 2000 and from that point forward occupies a significant
portion of the chart. Prior to this period, more generic descriptions are used.
Figure 13.1 Monthly volumes of PPs
wold lhl
105
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
14. The Horizon IT System Helpdesk was often the root of SLA
14.1.1
14.1.2
14.1.3
issues with POCL
The HSH was responsible for frontline support to users of the Horizon IT System.
Contractually, ICL Pathway’s SLA included items regarding the HSH, such as time to answer
a call and carrying through with a call (not abandoning it). The ICL Monthly Report discussed
the failings of the HSH, in regard to SLA requirements, for a significant amount of the review
period. Concerns were first raised in September 1998 and carried through the national
rollout. The same issues that triggered SLA concerns also “dented” confidence from POCL.
In May 2000, ICL Pathway declared an “own goal” based on HSH performance. The
management team was replaced, and improvements were noted in June 2000.
The following table contains verbatim extracts from the monthly reports (MRs) which I relied
on in identifying this theme. I have intentionally not made any corrections to grammar or
spelling. Where I deemed it helpful, I have highlighted certain sections in bold. The views
expressed in these extracts are that of the authors, being principally ICL Pathway, but in
some cases ICL Pathway and POCL.
Table 14.1 Verbatim extracts from Monthly Reports
URN
FUJ00058176
FUJ00058183
FUJO0058188
FUJ00058189
Title Date Extracted Text
Pathway Monthly September- The pressure on HSH to improve upon the 5 and 10
Report - September 98 minute call answering SLAs has been intensified.
1998 The dip in July's performance has been rectified but
both SLAs remain well below Minimum Acceptable
Level.
ICL Pathway Monthly —_June-99 The present performance of HSH gives cause for
Report - June 1999 concern in 3 areas:
Achievement of Service Levels.
Management Intervention.
Openness when dealing with ICL Pathway Service
Management.
ICL Pathway Monthly November- The SLAs being monitored are as follows:
Report - November 99 * Calls answered within 20 seconds.
1999
Measures for the Cash Account Period
(Wednesday/Thursday) have been revised
as follows:
« Availability of trained staff to answer a
Postmasters query.
* 100% of calls answered by a person trained in
using the cash account scripts.
. No more than 5% of calls must cause a ring-back
to the Postmaster due to the first level resource
exhausting their knowledge and a higher level
resource not being available.
* Where a ring-back is required as described
above it must occur within 20 Minutes.
ICL Pathway Monthly January-00 ‘The Corporate Red Alert with OSD for poor SLA
Report - January 2000 performance has been re-graded to Divisional Alert.
HSH Service improvements. are still necessary in
the areas of call answering and call abandonment
and second line support filtration rates.
106
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
FUJ00058190 ‘ICL Pathway Monthly —_February-
Report - February 00
2000
FUJ00058191 ICL Pathway Monthly — May-00
Report - May 2000
FUJ00058191 ICL Pathway Monthly May-00
Report - May 2000
FUJ00058192 ICL Pathway Monthly —_June-00
Report - June 2000
14.1.4
A number of the SLAs on the HSH are extremely
demanding, being beyond the normal industry
standards. Tony H prepared a document proposing
ways in which the service targets could be more
easily Met, and investigated the contractual issues
(Pathway to POCL, and Pathway to OSD). The
document sets out 18 recommendations for action.
Legal opinion is that although repeated failure to
meet the SLAs concerned will nor give grounds for
termination by POCL, it can hold our feet to the fire
and suspend the rollout.
POCL are shaping up to hit us on SLAs and
Training. This was predicted for about now on the
basis that, in the case of help desk metrics, we will
have failed to meet all criteria for three successive
quarters. That gives POCL the right to terminate
the contract. We don't expect them to want to do
that, but they can be expected to use the ‘default’
as a lever to force us to do better and make
concessions.
Weekly service performance is a key issue and
recent problems with Help Desk service have
significantly dented PO confidence. March and April
were disastrous months on OSD service levels,
driven by major resource issues (staffing levels) on
the Horizon System Help Desk. Nearly all of the
SLA's have been missed and significant penalties
incurred. This is an own goal and should have been
prevented. As reported last month, it is on Red
Alert and OSD have reacted decisively and
professionally to implement corrective action. Their
management has been changed and Over 40 new
help desk staff recruited along with a plan to recruit
at a pace to handle the weekly increase in Post
Offices and to cover for attrition. This has driven a
dramatic improvement- and this week we are now
back on target with 7 of the 10 key SLA's. The
sensitivity of this situation cannot be overstated. It
is highly visible and has brought firm reaction from
PO Directors. It will take week on week, month on
month good performance to recover our position.
As previously reported, weekly service performance
is a key issue and recent problems with Help Desk
service have significantly dented PO confidence.
However, I am pleased to report that OSD service
levels are now much improved and we are getting
back towards a reasonable SLA performance. The
poor service in Q1 has cost ICLI- over £200K in
penalties. It is intended to remove the Red Alert
within the next three weeks once we have
demonstrated consistent performance. It will take
week on week, month on month good performance
to fully recover the confidence of PO Directors.
Based on my understanding of the content of the PPs, I do not believe any of their content
would be relevant to ICL Pathway’s helpdesk concerns, and therefore I did not undertake
searches of the documents for this theme.
107
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
15. The Horizon SMC was frequently cited for not properly
filtering calls to the SSC. Thi:
SSC from resolving tech
15.1.1
15.1.2
15.1.3
15.1.4
15.1.5
lack of filte
al problems
ig delayed the
ICL Pathway had an error escalation process. Users that experienced problems with LHITS
were first directed to the HSH who logged the incident via PowerHelp. The SMC was
responsible for determining if the problem required the SSC to become involved. If the issue
was deemed necessary for escalation to the SSC it would then be recorded in the PinICL
system.
The SSC was responsible for the maintenance of PinICLs.
The ICL Monthly Reports discuss the topic of the SMC not properly filtering calls.
Consequently, the SSC was responsible for resolving an excess of PinICLs. The purpose of
the SSC was to resolve technical issues with the LHITS. The fact that the SMC did not filter
lower-level issues meant that the SSC was burdened with performing this triage. This extra
work delayed the SSC from addressing the true technical issues within the Horizon IT
System.
This problem persisted throughout the national rollout.
The following tables contain verbatim extracts from the monthly reports (MRs) and PinICLs
and PEAKs (PPs) which I relied on in identifying this theme. I have intentionally not made
any corrections to grammar or spelling. Where I deemed it helpful, I have highlighted certain
sections in bold. The views expressed in these extracts are that of the authors, being
principally ICL Pathway, but in some cases ICL Pathway and POCL.
Table 15.1 Verbatim extracts from Monthly Reports
URN
FUJO0058158
FUJ00058168
FUJ00058181
FUJ00058182
Title Date Extracted Text
ICL Pathway August-98 The Horizon Systems Helpdesk performance
Monthly Report — deteriorated over the past two months and is being
August 1998 monitored very closely. A number of corrective actions
are planned. This is part of a broader scrutiny of HSH
and SMC operation being undertaken in readiness for
full NR2 service.
ICL Pathway January-99 Actions to address underlying concerns with SMC
Monthly Report ~ performance have been identified and staff
January 1999 secondments are being arranged to aid skills transfer.
ICL Pathway April-99 The continuing failure of the SMC to adequately filter
Monthly Report - calls to SSC was escalated to Kevin Dowling (OSD
April 1999 Service Director). Kevin has promised an improvement
plan by mid May.
ICL Pathway May-99 SMC performance continues to be less than
Monthly Report ~ satisfactory. A service improvement plan has been
May 1999 produced and will be managed by Kevin Dowling.
Further secondments from the SMC to the SSC have
been identified.
108
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUJ00058186
FUJ00058187
Fuj00058188
FUJ00058189
Title
ICL Pathway
Monthly Report -
September 1999
ICL Pathway
Monthly Report -
October 1999
ICL Pathway
Monthly Report -
November 1999
ICL Pathway
Monthly Report -
January 2000
Date
September-
99
October-99
November-
99
January-00
Extracted Text
The workload pressure on SSC has intensified through
September with high volumes of calls being received.
SSC are struggling to contain the workload: the WIP is
in three figures. Also a number of additional tasks were
undertaken to support the resolution of Acceptance
Incidents. SSC had significant involvement in the
management and resolution activities for the
Correspondence Server index corruption problems that
occurred over the last week of the month. A major
factor in this is the overall performance of the SMC. A
pre-scan function is being put in place to ensure that
calls get the appropriate level of attention, but a
tougher stance will be taken on "invalid" calls passed
across by SMC. A RAR has been raised to recruit
additional resource.
The SSC remains under enormous workload pressure.
The number of calls received in October was 1123
(compared with 815 in September and 536 in August).
Poor filtration at SMC is a major contributor to this
problem and that aspect is part of the current OSD Red
Alert. A number of management and working level
meetings have been held with OSD and although a plan
to address the SMC failings is expected imminently
from OSD its appearance seems slow. As a temporary
measure overtime arrangements are being put in place
within the SSC to help handle the extra load.
Filtration at SMC remains a concern although the
implementation of a higher level of checking prior to
calls being escalated to SSC has certainly helped to
reduce the flow. A short secondment of one of the key
technical staff from the SMC to the SSC is also a
welcome move. A comprehensive plan for how OSD
expect to be able to improve the overall performance of
the SMC is still awaited.
The Corporate Red Alert with OSD for poor SLA
performance has been re-graded to Divisional Alert.
HSH Service improvements. are still necessary in the
areas of call answering and call abandonment and
second line support filtration rates.
Table 15.2 SSC management information
URN
FUuJ00058183
Fuj00058184
FUJ00058185
FUJO0058186
FUJ00058187
Document Title Calls Calls Calls closed by
raised closed SSC as Known
through through Error/Duplicat
ssc ssc Call/No Fault
in Product
ICL Pathway Monthly Report - June 410 498 158
1999
ICL Pathway Monthly Report - July 496 427 124
1999
ICL Pathway Monthly Report - August 536 529 159
1999
ICL Pathway Monthly Report - 815 737 N/A
September 1999
ICL Pathway Monthly Report - October 1123 1070 639
1999
109
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
EXPG0000001
EXPG0000001
URN Document Title Calls Calls Calls closed by
raised closed SSC as Known
through through Error/Duplicat
ssc ssc e Call/No Fault
in Product
FUJO0058188 ICL Pathway Monthly Report - 865 1068 510
November 1999
FUJO0058190 ICL Pathway Monthly Report - 1555 1677 536
February 2000
FUJ00058191 ICL Pathway Monthly Report - May 1595 1668 552
2000
FUJ00058192 ICL Pathway Monthly Report - June 1531 1662 1067
2000
FUJO0058193 ICL Pathway Monthly Report - July 1293 1662 1040
2000
FUJO0058194 ICL Pathway Monthly Report - August 721 853 413
2000
FUJ00058195 ICL Pathway Monthly Report - October 987 1136 532
2000
FUJO0058196 ICL Pathway Monthly Report- 1244 1411 684
November 2000
FUj00058197 ICL Pathway Monthly Report - 812 884 383
December 2000
15.1.6 I Commentary about misdirected calls from the SMC to the SSC are also captured in the PPs.
Table 15.3 Verbatim extracts from Monthly Reports
URN Ticket Date Extracted Text
Source
FUJ00027211 PinICL 27/05/1999 "The previous text in this call states - Andrew at ITSA suggested we
return this call so that PO info can be added and the correct
referance no can be supplyed. WHY WAS THE CALL NOT RETURNED
TO ITSA ? Contacted Emma at ITSA. The ITSA ref is incorrect.
Suggest we keep call open until ITSA chase HSH - THEN WHAT ?
ITSA have suggested that the call be retunred to them in order for
them to add information which was necessary for the diagnosis -
why was the call sent to the SSC ?"
FUJ00027003 PinICL 17/06/1999 "This problem has already been investigated. It says in the KEL:
"This will be fixed in LT2 (see pc24986)." The advice for the PM is
also included in the KEL as has already been noted: "The figures for
the following week will not be affected." I am unsure why this
was sent to the SSC."
FUJO0030450 PinICL 13/10/1999 "I do not understand why this call has been sent to SSC.
There was a comms problem, this was apparently sorted out by
CFM. SMC has confirmed that the health checks on all counters
were OK. What is the problem now?”
FUJ00032423 PinICL 11/11/1999 “Why has this call been sent to SSC?"
FUJO0043195_— PinICL 19/05/2000 "SMC1 Information: sent to SSC in error - please send back
over OTI when it appears - Thanks.
FUJ00062974 PEAK = 12/07/2000 "Called PM who says user is no longer locked, therefore no action is
necessary. I assume user logged back onto original counter where
report was processed as per kel, Richardi0.htm PM happy to close
call. (...Why was call sent to SSC, and given that call was sent,
why did it take 6 days???)"
110
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUJ00066464
Fuj00068089
FUJ00073008
Ticket
Source
PEAK
PEAK
PEAK
Date
25/08/2000
25/09/2000
13/12/2000
Extracted Text
"Unable to ping this SCO. Call should not have been sent to
SSC, according to the new non-polling procedures the next stage is
to send an engineer to site.”
"PRESCAN:Before the call was sent to us, it was updated as follows:
-25/09/00 11:42 GB082641 Information: This error relates to KEL
Reference: PCarroll1535R.htm , which states that the Patch should
be regressed and re-applied. The counter should NOT be swapped.
This is not a Cl4 site. Why has the call been sent to SSC? I
thought that SMC was tasked with the patching process.”
"From the call log it seems that the transactions in question had
been done on counter 4, The update on 11/12 at 15:42 says that
counter 4 was not comunicating. This would cause the transactions
that had been done on counter 4 to be 'missing' on the reports done
on any of the other counters. The call text also says that the rmn
has recified the situation. Therefore I am not sure why this call
has been sent to SSC."
111
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
16. The SSC was overwhelmed with PPs but was earnest in its
effort to perform its duties
16.1.3
16.1.4
The SSC was responsible for the resolution of PPs. The SSC also maintained the KELs. The
SSC was tasked with resolving PPs to the best of its ability and then communicating
resolutions to the SMC.
The ICL Monthly Reports often call attention to the workload of the SSC.
Over the course of reviewing the PPs provided to me for this Report, I recognized the
complexity of some of the issues the SSC was responsible for resolving. Throughout the
course of my review, most of the entries I read appeared to be written by SSC staff and
demonstrated to me that they were earnest in their efforts to resolve these issues.
The following table contains verbatim extracts from the monthly reports (MRs) which I relied
on in identifying this theme. I have intentionally not made any corrections to grammar or
spelling. Where I deemed it helpful, I have highlighted certain sections in bold. The views
expressed in these extracts are that of the authors, being principally ICL Pathway, but in
some cases ICL Pathway and POCL.
Table 16.1 Verbatim extracts from Monthly Reports
URN
FUJ00058166
FUJ00058177
FUJ00058182
Title Date Extracted Text
Pathway Monthly Report - December- In December, 213 PinICLs were raised.
December 1997 97 103 PinICLs were closed by the SSC and
74 transferred to Development for
resolution. This must be seen as a serious
distraction to the development teams
who, are focused on Release 2.
Pathway Monthly Report - October-98 Model Office is providing a very high
October 1998 workload for the SSC and I am concerned
that NR2 software quality should show
significant improvement for MOT in order
that we can achieve Release
Authorisation.
ICL Pathway Monthly Report - May-99 SSC staff have been under considerable
May 1999 workload pressure from the Data Centre
Migration weekend onwards. They have
been called out most nights to deal with
system problems although most of these
have been repeat problems rather than
lots of new issues. The PinICL stack has
regularly exceeded 100 calls.
112
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN Title
FUJO0058186 ICL Pathway Monthly Report -
September 1999
FUJO0058186 ICL Pathway Monthly Report -
September 1999
FUJ00058187 ICL Pathway Monthly Report -
October 1999
FUJO0058187 ICL Pathway Monthly Report -
October 1999
Date
September-
99
September-
99
October-99
October-99
Extracted Text
The workload pressure on SSC has
intensified through September with high
volumes of calls being received. SSC are
struggling to contain the workload: the
WIP is in three figures. Also a number of
additional tasks were undertaken to
support the resolution of Acceptance
Incidents. SSC had significant
involvement in the management and
resolution activities for the
Correspondence Server index corruption
problems that occurred over the last
week-of the month. A major factor in this
is the overall performance of the SMC. A
pre-scan function is being put in place to
ensure that calls get the appropriate level
of attention, but a tougher stance will be
taken on "invalid" calls passed across by
SMC. A RAR has been raised to recruit
additional resource.
The workload pressure on SSC has
intensified through September with high
volumes of calls being received. SSC are
struggling to contain the workload: the
WIP is in three figures. A major factor in
this is the overall performance of the
SMC.
The SSC remains under enormous
workload pressure. The number of calls
received in October was 1123 (compared
with 815 in September and 536 in
August). Poor filtration at SMC is a major
contributor to this problem and that
aspect is part of the current OSD Red
Alert. A number of management and
working level meetings have been held
with OSD and although a plan to address
the SMC failings is expected imminently
from OSD its appearance seems slow. As
a temporary measure overtime
arrangements are being put in place
within the SSC to help handle the extra
load.
The number of special Acceptance
Incident related reports that the SSC has
had to produce has reduced somewhat
this month. However, the administrative
overhead of handling calls raised, as a
result of the Non-polled Offices Report
remained significant.
113
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUJ00058188
Fuj00058194
FUJ00058196
16.1.5
Title
ICL Pathway Monthly Report -
November 1999
ICL Pathway Monthly Report -
August 2000
ICL Pathway Monthly Report-
November 2000
Date
November-
99
August-00
November-
00
Extracted Text
There was some slight relief in the
workload pressure on the SSC,
particularly over the last ten days of the
month. The number of calls received in
November was 865 compared with 1123
in October. SMC filtration remains a
concern although implementation of a
higher level of checking, prior to calls
being escalated to SSC, has certainly
helped to reduce the flow. SSC
successfully achieved the clearance of the
outstanding Counter index corruptions at
some 300 Outlets a commendable
achievement in co-operation with design
and development.
The SSC have been heavily involved in
dealing with Correspondence Server
related issues, the stability of which has
been causing concern. The Wigan
Bootserver is now more stable; following
a significant reduction by Energis of
incorrectly routed "rogue" calls from live
Outlets.
There have been a number of issues on a
small proportion of Counters as they
migrate from CI_3 to Cl_4, which has
resulted in a high urgent workload for
SSC staff.
The following figure’s line shows the open balance of PPs by day. There are maximums in
the autumn of 1997 and the autumn of 1998. There is a noticeable downward trend that
produces a minimum in July 1999, followed by a steep rise, cresting in May 2000. This is
followed by a moderate reduction through September 2000. The remainder of the data
shows another rise until the end of 2000, which is where the PP production ends. The
remainder of the figure’s line shows how this PP balance is fully closed in November 2002.
For the review period, the average open PP balance was almost 1,400. In my opinion, this
is a high amount.
114
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
3,500
3,000
2,500
2,000
1,500
1,000
500
Figure 16.1 Open PP Balance By Day
on
aus
aap
ese
s286
g< 9
5 8 8
235
ao
o oO
Ss 8
8
16.1.6
966T Jequieceq TO
266T YOLEW TT
Z66T eunt 6T
L66T Jequiexdes £7
866T Aenuer so
866T [dy ST
866T AINE bz
866T JequeAoN TO
666T Areniqa4 60
666T ACW OZ
666T asnbny gz
666T 1equiaseq 90
0002 Yo1eW ST
000z eunc €z
The persistent high balance was a combination of new PPs being raised and the amount of
0002 4840390 To
Tooz Asenuer 60
To0z Idy 6T
Tooz Aine sz
TOOZ 4@qWEAON SO
Zo00z Avenigas ET
7002 AeW bz
Z00z Jequiaydas To
Z00z 4equis29q OT
€00Z YEW OZ
time it took to resolve existing PPs. The following figure shows a distribution of the amount
of time to fully close PPs. On average, 43 days were required to close a PP, with almost
3,000 PPs requiring more than 180 days to resolve.
115
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
35,000
30,000
25,000
20,000
15,000
10,000
5,000
Figure 16.2 Days to resolve PPs
(0, 20]
Days to Resolve
(20, 40] (60, 80) (100, 120] (140, 160] > 180
(40, 60] (80, 100] (120, 140] (160, 180]
T identified 48 individuals that were involved in the resolution of at least 1,000 PPs. Between
these 48 individuals, 50,983 PPs were resolved. This represents 90.25% of all PPs in my
review set.
Table 16.2 Users involved in PPs
PP Count User Name
17,357 Barbara Longley
11,377 Lionel Higman
7,382 Patricia McLoughlin
4,412 Kevin Barrett
3,894 Richard Coleman
3,189 John Simpkins
3,027 Diane Rowe
2,988 Les Ong
2,909 Nikki O'Sullivan
2,584 Paul Steed
2,535 John Moran
2,533 Eric Jennings
2,506 Shehbaz Ziauddin
2,480 Mike Holms-Sharp
2,372 Mike Croshaw
2,250 John McLean
2,168 Pat Carroll
2,133 Steve Warwick
116
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
PP Count User Name
2,045 Catherine Obeng
2,023 Glynne Rogers
1,822 Peter Morgan
1,801 Nam Pandher
1,789 Steve Parker
1,756 John Budworth
1,751 Mark Wright
1,743 Angela Shaw
1,721 Tim Canniffe
1,701 Walter Wright
1,652 Dave Colclough
1,623 Deirdre Conniss
1,610 Ajay Nehra
1,563 Rakesh Patel
1,403 Ken Wood
1,400 Kevin McKeown
1,362 Jim Anscomb
1,351 Doug Jones
1,299 Cliff Sawdy
1,149 Dave Royle
1,145 Garrett Simpson
1,126 Denise Jackson
1,121 Asim Mushtaq
1,108 Phil Hemingway
1,026 Miho Fujii
1,020 Anna Croft
1,020 Dao Ly
1,019 Peter Jobson
1,016 Bill Hillyard
1,000 Michael Howell
117
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
17.
Acceptance In
success of ICL Pathway. A per:
376.
i744
17.1.2
17.1.3
17.1.4
17.1.5
17.1.6
17.1.7
17.1.8
17.1.9
17.1.10
lents were a gating issue to the financial
ting issue related to AI
Acceptance was the term used by ICL Pathway and POCL to indicate the Horizon IT System
operated in a manner that was acceptable by POCL. Acceptance Incidents ("AIs”) were
identified shortcomings of the Horizon IT System that required resolution prior to Acceptance
being confirmed by POCL.
Acceptance was financially significant to ICL Pathway. ICL Pathway was paid once
Acceptance was achieved. It received a high degree of attention by ICL Pathway.
The Monthly Reports describe existing Als, and ICL Pathway's efforts resolve them. AI 376
(Accounting Integrity) caught my attention. Accounting integrity is a fundamental
requirement of the LHITS. AI 376 was one of the final Als to be closed.
24 September 1999 marked the day Acceptance was granted, triggering a £68m invoice
delivered to POCL on 27 September 1999 to be paid in 30 days.
In November 1999, at least one full month and possibly two full months after acceptance
was granted, ICL Pathway reported that "POCL have come round to the understanding that
dealing with residual AI 376 concerns in the short to medium term will rely on processes
and tools but no new software features as such.”
In January 2000, ICL Pathway states “If pressed POCL would agree that Als 342, 372, 376,
378, 218, 391 are Closed / incapable of further update. Their Acceptance Manager is leaving
the project at the end of February.” Further in the same report it states “The outturn on
AI376 was 0.06% Cash Account Discrepancies, exactly an order of magnitude better than
the target. Under this activity John P made significant contributions to the Third
Supplemental agreement, specified the committed CS Repair Facility, aligned the operating
agreement on Reconciliation to support the contract, and sorted out the necessary PinICLs
to clear.”
In February 2000, ICL Pathway declared that the POCL Acceptance Manager has left the
project and transferred the residual actions to “business-as-usual”.
It is unclear to me what exactly took place to close AI 376. The reading of these entries
leaves much room for interpretation.
Regardless, the fact that accounting integrity was a persistent issue in the national rollout
of the LHITS cannot have been the intention of the sponsors nor the goal of ICL Pathway.
The following tables contain verbatim extracts from the monthly reports (MRs) and PinICLs
and PEAKs (PPs) which I relied on in identifying this theme. I have intentionally not made
any corrections to grammar or spelling. Where I deemed it helpful, I have highlighted certain
sections in bold. The views expressed in these extracts are that of the authors, being
principally ICL Pathway, but in some cases ICL Pathway and POCL.
118
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
EXPG0000001
EXPG0000001
Expert Witness Report of Charles Cipione, dated 14 September 2022
Table 17.1 Verbatim extracts from Monthly Reports
URN Title Date
FUJO0058168 Pathway Monthly January-99
Report -January 1999
FUJ00058182 ICL Pathway Monthly May-99
Report - May 1999
FUJO0058183 ICL Pathway Monthly —_June-99
Report - June 1999
FUJO0058183 ICL Pathway Monthly June-99
Report - June 1999
FUJ00058184 ICL Pathway Monthly July-99
Report - July 1999
FUJO0058184 ICL Pathway Monthly —_July-99.
Report - July 1999
FUJ00058184 ICL Pathway Monthly July-99
Report - July 1999
Extracted Text
Management has bought into the need to find new
ways to reduce costs. The immediate problem is that,
in the short term, the pressures to achieve
Acceptance and the required rate of roll out continue
to drive demands up not down. The challenge right
now is to hold to the current net spend forecast
(after revenue recoveries) and at the same time hold
to Programme milestones: Cost Down measures will
be required to achieve this.
CASH FLOW
VERY SENSITIVE TO DELAY
* Acceptance = £68m
* — 1800 post offices (300 +1500) = £90m
* 2 month slip = > £10m - £100m impact in
1999/2000
* NB. Peak cash goes from £370m to £470m
* vs. ICL Treasury assumption of £420m
We are determined to meet the cash payment points
which follow Acceptance (£68m) and successful Roll-
out to the first 1,800 Post Offices (£90m) which are
vital payment points in 1999 both for ICL/Fujitsu
funding and of course for the credibility of the new
programme moving forward. Alll staff are focused on
the criticality of meeting these milestones.
Banking arrangements remain to be restructured
satisfactorily bridging finance from Fujitsu may be
required to provide the short term cash required pre
Acceptance and possibly pre the first £90m progress
payment. Stringent cash controls are in place. The
subcontractor termination negotiations have taken
account of the need to defer cash payments.
The number of Acceptance Incidents POCL raised to
high (plus medium tending to high) presents a real
difficulty in completing the Acceptance Process to
time (i.e. a decision on 18 August). The whole team
are re-doubling their efforts to achieve this.
Problems still exist in Live Trial operation where over
50 "printer hang" problems are being raised weekly
and System "freezes" are occurring. To clear these
Postmasters are rebooting, sometimes without first
contacting the HSH, which impacts POCL's service to
its customer. This is now the most critical issue and
has resulted in the Acceptance Incident relating to
system stability being raised to high (AI298).
Analysis of incidents, both reported and not reported,
is in hand along with investigation of potential root
causes.
We now have to work with no overdraft facilities at
all, and cash flows up to the receipt of the first
acceptance payment and in the voids between
subsequent stage payments will be tightly rationed
and monitored.
119
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
Fuj00058185
Fujo0058185
FUJ00058185
FUJO0058185
FUJO0058186
FUJ00058186
FUJO0058186
FUJO00058186
Title
ICL Pathway Monthly
Report - August 1999
ICL Pathway Monthly
Report - August 1999
ICL Pathway Monthly
Report - August 1999
ICL Pathway Monthly
Report - August 1999
ICL Pathway Monthly
Report - September
1999
ICL Pathway Monthly
Report - September
1999
ICL Pathway Monthly
Report - September
1999
ICL Pathway Monthly
Report - September
1999
Date
August-99
August-99
August-99
August-99
September-99
September-99
September-99
September-99
EXPG0000001
EXPG0000001
Extracted Text
The speed of turnaround required for acceptance
incidents is straining the Release Management
processes to the limit, however the software
distribution mechanism continues to operate well
with distributions successfully committing to
approximately 780 (out of 820) counters on the first
pass.
211: Receipts and Payments not equal in the double
entry. Although this Al was cleared there has been
some regression and not all of the new incidents are
yet fixed.
As anticipated last month, the problems experienced
by the live trial outlets with the Epson back office
printer ‘hanging’ during the production of the weekly
cash account became a serious acceptance incident
which is proving extremely difficult to resolve.
OpCo now has to live without an overdraft facility,
and until the acceptance payment is received from
POCL, there will be partial reliance on ICL Group to
settle intercompany liabilities on our behalf.
SMC's performance in catching up on delivering
outstanding fixes to newly-installed counters and to
replacement counters is unsatisfactory and this is
likely to cause problems during the Acceptance
monitoring period. Proposals are therefore being
worked on to install all fixes to such counters at the
time of their installation/replacement.
The Acceptance Resolution Timetable contains some
300 activities and events that are pacing items for
the restart of National Roll-out on 24 January 2000.
Amongst these are performance measures relating to
Acceptance Incident 298 (System Stability), 376
(Accounting Integrity) and 408 (Help Desk) which will
be monitored during October and the first half of
November and reviewed to see if the criteria have
been achieved on 24 November.
Acceptance was achieved on 24 September and
the resultant invoice for £68m delivered on 27
September for payment within 30 days.
Acceptance was achieved on 24 September triggering
the 1999 element of National Roll-out. The Roll-out
ramp up has progressed better than expected given
the potential impact of compressed time for the
processes and the start stop nature of decision
making resulting from the difficulties experienced
during the Acceptance Process. As at the 10 October
978 offices had been installed and migrated in total
with 199 installations being completed the previous
week.
120
Post Office Horizon IT Inquiry
EXPG0000001
EXPG0000001
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN Title
FUJO0058186 ICL Pathway Monthly
Report - September
1999
FUJ00058186 ICL Pathway Monthly
Report - September
1999
FUJO0058187 ICL Pathway Monthly
Report - October 1999
FUJ00058187 ICL Pathway Monthly
Report - October 1999
FUJ00058187 ICL Pathway Monthly
Report - October 1999
Date
September-99
September-99
October-99
October-99
October-99
Extracted Text
The expansion of the live estate has meant that the
number of outlets not returning transaction details to
TP, due to ISDN problems or simply that the terminal
is powered down, has increased. This is becoming a
job in itself to track and resolve. We are obliged
under the rectification plan for AI376 to raise an
incident on each office that hasn't polled. This is time
consuming and probably pointless for those offices
only down for 24 hours. Richard Brunskill is due to
talk to the customer (with the Requirements team) to
try and find a more efficient way of tackling this
problem.
A Second Supplemental Agreement resulted from
Acceptance and introduces an optional 1600
milestone for National Roll-out prior to Christmas.
This yields a £80m payment, with £10m held over to
the next milestone (May), but we would still get the
full £90m in December if we were to achieve 1800.
In addition this Agreement introduced an Acceptance
Resolution Timetable into the contract.
298: (Tony H and Dave H) The four week observation
period will start on 21/10. (CCN555 has been raised
to make the observation Cash Account Week
integral.) All fixes are available and a tracking
document to record progress set up. On the cut off
date of 1/10 the test sample was established as 782
eligible rolled-out outlets representing 1777 eligible
counters. The target is a figure of merit of four units
per counter per year, a unit being an authorised
reboot or various numbers of workaround. The CAP
28 figure result was around five units on a very good
trend. For CAP29 the result rose to around seven
units because of 376-type issues (see above), new
offices not being brought up to current software
revision levels immediately before first use and some
offices not yet equipped with fixes for printer
incidents.
The monitoring of the three big Acceptance Incidents
(AI298, A1376 and AI408) have all run into
difficulties in varying degrees with the common
theme being the potentially unsafe state of operation
of Reference Data within the end-to- end model. As
mentioned already the end-to-end workshop is the
critical process for finding an acceptable resolution to
this complex area.
It is essential that the estate software revision levels
are complete and operations stabilised, such that 376
and 218-type incidents are minimised.
121
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUJ00058187
FUJ00058187
FUJ00058187
FUJ00058188
FUJ00058188
Title
ICL Pathway Monthly
Report - October 1999
ICL Pathway Monthly
Report - October 1999
ICL Pathway Monthly
Report - October 1999
ICL Pathway Monthly
Report - November
1999
ICL Pathway Monthly
Report - November
1999
Date
October-99
October-99
October-99
November-99
November-99
EXPG0000001
EXPG0000001
Extracted Text
376: (John P) This area is of particular concern. The
six-week observation period has started. The work is
in three parts: fixes yielding a target stability figure
of merit of a maximum 0.6% of Cash Accounts in
error (approximately 42); additional reconciliation
facilities; and new Operational Business Change
(OBC) procedures. Although all fixes are
implemented, problems arising from Pathway
Reference Data handling were encountered and are
proving difficult to solve without letting through Cash
Accounts in error. The definition work for additional
reconciliation is on plan and design is in progress. All
the OBC procedure work is completed. The POCL
Acceptance Test Manager has left the project and
several new people are now involved and are not yet
familiarised.
Too many reference data errors are being distributed
to the counter. End to end design reviews are being
held to establish what action can be taken swiftly to
prevent these occurring in the future. These are
having a major impact on Acceptance Incident 376.
In addition, the performance of the data distribution
process is inadequate and must be improved before
roll-out commences in late January 2000.
Managing the Acceptance Resolution Plan during the
balance of 1999 will be critical to our clean start in
year 2000. Within this the Reference Data end-to-
end concerns are the most important and do require
a positive joint attitude from POCL as well as careful
planning and defensive mechanisms from within ICL
Pathway.
The focus of work has remained the resolution of
residual Acceptance incident hurdles such that both
POCL and Pathway are happy for roll out to re-
commence on 24th January. POCL have confirmed
that that is still their intention subject to being
satisfied as to progress on AI 376 in particular.
The most serious issue on acceptance resolution
concerns AI376 and the integrity of accounting data
being managed from the end to end basis with
Horizon. This in turn requires more disciplined and
strict accounting integrity controls, some of which
can be achieved through the EPOSS reconciliation
software and others through process and
independent tools and the balance through stronger
end to end control of the reference data processes.
The plan to handle the main problem area and indeed
the lower level actions across a range of Al issues is
well constructed, being followed and is capable of
achievement. However, there is little contingency in
the plan with respect to timescale and we do need a
formal agreement with POCL, particularly in the case
of reference data procedures which have been
defined during workshops. These need to be drawn
together into an agreed Change Control document,
probably a further Supplemental Agreement.
122
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN Title Date Extracted Text
FUJO0058188 ICL Pathway Monthly November-99 A148 for Help Desk procedures is a second but
Report - November important acceptance resolution plan, where we now
1999 have a new set of measures for calls answered within
20 seconds, cash account response times and cash
account script compliance. This is largely a subset of
the original SLA measures and will be reviewed on a
weekly basis between the 3" December to mid
January.
FUJO0058188 ICL Pathway Monthly November-99 _As part of the AI376 rectification plan, MSU
Report - November presented to POCL TIP the incident management
1999 process for business critical incidents raised by POCL
or via the newly developed EPOSS exception report
set. Initial comments received from TIP were
favourable and they applauded the tighter
management controls that ICL Pathway is
introducing. Non-polled offices are still creating a
large number of incidents. MSU are identifying where
there is a specific system problem preventing the
Outlet from polling. However there is still the
problem where MSU suspect that Outlets are turning
the Counter equipment off - evident from Mondays
reports which contain 3 to 4 times the number of
non-polled Outlets than other days within the week.
FUJO0058188 ICL Pathway Monthly November-99 _As part of the A1376 rectification plan, MSU
Report - November presented to POCL TIP the incident management
1999 process for business critical incidents. Initial
comments from TIP were favourable and they
applauded the tighter management controls that ICL
Pathway is introducing.
123
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
FUJO0058188 _ ICL Pathway Monthly November-99 ACCEPTANCE RESOLUTION TIMETABLE Progress
Report - November against the 13 Acceptance Incidents, forming the
1999 core of the Acceptance Resolution Timetable, is
reviewed below in Acceptance Resolution order: A
Red Flag issue is that TIP has apparently not
developed the manual input facility for low volume
corrections.
There is a short list of PinICLs that we must deal with
to clear up the Acceptance Resolution Timetable and
make it possible for POCL to approve clearance of
individual elements. This will be circulated
separately.
‘A meeting to establish status and actions on 211,
342, 376 and 378 was held 7/12 with the POCL
Acceptance Test Manager.
211: POCL commented to ICL Pathway on 1/12 that
mismatches continued to occur for a number of
reasons, that it required. Payments and Receipts
always to be in sync and that the incident was not
ready for closure. A more detailed description of
POCL's intent was provided on 30/11, in effect asking
for Payments and Receipts to be forced equal
through some form of automated suspense account
entry.
The only current instances of Payments and Receipts
being unequal are where the Horizon system has to
inherit a manual or ECCO system that was
unbalanced before migration.
The issue was discussed on 2/12 in a full forum and
POCL reconfirmed its position that such unbalanced
migrations were preferable to making an automated
suspense account transfer.
Closure continues to be sought.
342: POCL commented to ICL Pathway on 1/12 that
TIP references 986 and 995 had occurred and were
awaiting analysis and rectification plans.
In the case of reference 986 the office was on an
extended CAP, which should have been known to
POCL. Reference 995 was caused by the outlet
having fewer counters installed than were previously
scheduled and the system controls will not initiate
polling until all counters have participated in a first
end of day. In summary, there is no problem with
ICL Pathway software.
Closure continues to be sought.
390: The APS recovery software enhancement was
distributed 29/11 and monitoring is now in progress
as scheduled.
376: The POCL action to approve clearance of all
incremental fixes installed by 14/9 from field
evidence remains open.
The Pathway procedures for manual input were
presented to TIP/TP on 1/12 and material for the test
of manual input has been prepared. However, POCL
has not performed activity 376.371 "POCL Develop
and Test manual input facility" and so this is now a
Red Flag issue.
Cycle 3 testing of the Additional EPOSS Reconciliation
Testing started on 3/12 and should complete 10/12.
Software distribution is scheduled for 17/12 about 10
days ahead of plan.
124
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
The measurements will continue for the period 2/12
to 12/1 and will exclude items that would have been
prevented by a range of additional Reference Data
controls being introduced before the rollout.
378: Improved diagnostic/defensive code has been
produced and will be distributed at the same time as
the Additional Reconciliation. There are no instances
yet trapped.
369: Further tests following the ‘Pilot’ scheme were
on schedule to complete 8/12. POCL is producing a
report, which it will review with DSS on 22/12,
including an analysis of impounded OBCS books and
logistics of re-supply. ICL Pathway understands that
POCL will request DSS to close this incident.
There were 119 impounds in the first three weeks.
ICL Pathway understands this is a slightly lower rate
than during the first pilot and that analysis of books
by PIRA shows the quality of the offending printed
bar-codes to be still poor.
POCL commented to ICL Pathway on 1/12 that it
required an update on Pathway's investigations into
the problem of bar-code reading after a manual
scales transaction. An update was provided at the
Delivery Meeting of 24/11. A PinICL to resolve the
issue of scanning after manual scales is in progress.
A simple workaround is available. POCL also stated
on 1/12 that Pathway had failed to produce statistics
for the first two weeks of the four-week trial as.
agreed. In fact ICL Pathway had delivered the
statistics POCL had asked for covering the first two
weeks on 30/11. POCL has since asked for a daily
level of analysis. The final two weeks totals will be
available in a few days time - the daily OBCS.
transaction counts will also be provided.
John C provided POCL with a paper, which was well
received, describing potential improvements to the
OBCS scanner/bar-coding system.
372: A review of the distributions for Riposte 5.4.10
and EPOSS.3_20 rollup was conducted successfully.
A final report on software distribution was provided
1/12.
298: The target was that there should be no more
than 560 qualifying incidents between CAP31 and
CAP34. In fact, there were 551.5 qualifying incidents,
including a spike of Blue Screen incidents associated
with a major switch failure within the Energis
network. The weekly incidents since the end of the
monitoring period have remained below the
equivalent weekly level of 140 incidents per week
(89.5 for CAP35, 114.5 for CAP36, 94 for CAP37).
The review of Revisions to the Testing & Integration
Approach for Pathway Release C$R+ was completed
successfully.
Closure continues to be sought.
218: All Pathway’s actions were confirmed by POCL
as completed, on 22/11. There are still some POCL
actions not yet complete. The Performance Review
Report has been produced.
There are two open Low incidents (364 & 365) linked
to Al 218, relating to training mode and these may
hamper closure of 218 itself.
CCN566a (Training Window) was approved 3/12.
125
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
Fuj00058188
Fuj00058188
Fuj00058188
FUJ00058188
Fuj00058189
Title
ICL Pathway Monthly
Report - November
1999
ICL Pathway Monthly
Report - November
1999
ICL Pathway Monthly
Report - November
1999
ICL Pathway Monthly
Report - November
1999
ICL Pathway Monthly
Report - January 2000
Date
November-99
November-99
November-99
November-99
January-00
EXPG0000001
EXPG0000001
Extracted Text
POCL Post installation processes have been defined in
Operational Performance Assessment, post
installation questionnaires and training mode
documents and processes produced. These will be
implemented from the restart of rollout.
391: The Wigan exclusion zone fence structure is
complete and the alarm system installation is
forecast 10/12, two weeks or so earlier than plan.
Installation of the card access controls for the Wigan
back gate is now complete. Fire regulation issues
have hampered installation of the new palisade fence
at Bootle: this is forecast to complete 17/12.
314: The draft document (without Appendices) was
completed ahead of schedule 22/11 and formally first
reviewed on schedule 1/12. POCL wants to extend
the reviews beyond the period planned. We have
notified them that such extended reviews must be
accommodated within the overall schedule.
408: A new set of measures for calls answered within
20 seconds, cash account first and second line
responses, and cash account script compliance have
been agreed for the period 3/12 to 13/1.
412: Closure continues to be sought.
The "big three" Acceptance issues have been reduced
to the "big two" with the clearance of 298 (Counter
Stability). Actions have been developed for handling
the issues on EPOSS Reconciliation and Reference
Data sufficient to get us to the decision to restart the
rollout on 24 January. There are new starts on
Network Banking and Euro study.
POCL decided not to stop roll out on 24th November
or ten days later having explored with us the way
forward on outstanding Al issues.
POCL have come round to the understanding
that dealing with residual AI376 concerns in
the short to medium term dual will rely on
processes and tools but no new software
features as such.
We now move on to the EPOSS reconciliation facility,
which is required for AI376 and is one of the other
critical criteria to be reached to enable National
Rollout to restart. This is a much more complex
facility than SIPI6 and needs to be satisfactorily and
safely delivered to all the live post offices before the
28 December such that we can have two clear
weeks of cash account running to ensure accuracy,
stability and effectiveness. The plan is tight,
manageable but not without risk.
ACCEPTANCE LOOSE ENDS If pressed POCL would
agree that Als 342, 372, 376, 378, 218, 391 are
Closed / incapable of further update. Their
Acceptance Manager is leaving the project at the end
of February. The formal timetable was updated, and
we are down to minor points. The formal
measurements for Als 376, 408 and 298 continued
until the end of CAP42 (12/1) and are now
completed.
126
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN
FUujJ00058189
FUJ00058189
FUJ00058189
FUJ00058189
FUJ00058189
FUJ00058190
FUJO0058190
Title
ICL Pathway Monthly
Report - January 2000
ICL Pathway Monthly
Report - January 2000
ICL Pathway Monthly
Report - January 2000
ICL Pathway Monthly
Report - January 2000
ICL Pathway Monthly
Report - January 2000
ICL Pathway Monthly
Report - February 2000
ICL Pathway Monthly
Report - February 2000
Date
January-00
January-00
January-00
January-00
January-00
February-00
February-00
EXPG0000001
EXPG0000001
Extracted Text
NEW BUSINESS Now that Acceptance has been
achieved and National Roll-out continues, there are
signs that discussions will take place again on
developing the service. The main forward move is a
joint team meeting with the Post Office Network on
8" February. The principal aims of this event are to
lay the ghosts of the past to rest and to develop a
more positive approach to the future, specifically we
want to establish an MD level Steering Board. We
have been warned by PO to expect a difficult
meeting.
The outturn on AI376 was 0.06% Cash Account
Discrepancies, exactly an order of magnitude better
than the target. Under this activity John P made
significant contributions to the Third Supplemental
agreement, specified the committed CS Repair
Facility, aligned the operating agreement on
Reconciliation to support the contract, and sorted out
the necessary PinICLs to clear.
Discussions to change to the migration utilities and
EPOSS for force-balancing under Al 211, Receipts not
equal to Payments, have continued to the present-
day and now require a paid study before the CRs can
be raised.
MSU has been working successfully with POCL to
close down long outstanding PinICLs and issue final
versions of all outstanding RED reports. The team is
now getting ready for the introduction of new
incident management procedures following the
resolution of AI376.
Since our last report, much of CS energy has been
directed at addressing Acceptance and resolving
problems with the Reference Data interface to PO.
There is, of course, much sweep-up work on
Acceptance and CSR and CSR+.
ACCEPTANCE LOOSE ENDS The POCL Acceptance
Manager has now left the project and handed
over the residual actions to business-as-usual.
We have dealt with queries from POCL concerning
AI376. One formal letter has been responded to
attempting to avoid the conclusion that we had not
found EPOSS reconciliation incidents that we should
have found or that we have not reported those we
did find, In reality CS are greatly hampered in
"spotting the incident" because the reports have not
had fixes implemented and report large amounts of
do-nothing information. We have attended the
Release Management Forum and proposed some re-
ordering of the fix backlog, but it will be at least until
the first week of March before this situation
improves. Also the requirements of security have
caused reports to be retrieved manually rather than
by automated mail and handling mistakes are
inevitable. In addition some changes to the CS
procedures on Reconciliation have been devised. The
CP to provide CS with a TIP file Repair Facility is
provisionally approved pending OTT impact.
Extensions to it sought by CS will be the subject of
separate CPs.
127
Post Office Horizon IT Inquiry
EXPG0000001
EXPG0000001
Expert Witness Report of Charles Cipione, dated 14 September 2022
17.1.11 Acceptance and the related acceptance incidents directly impacted ICL Pathway’s financial
goals. As such, SSC team members acknowledged the importance of PPs related to
acceptance incidents.
Table 17.2 Verbatim extracts from PPs
URN Ticket Date
Source
FUJ00029789 PinICL 30/06/1999
FUJ00029832 PinICL 12/08/1999
FUJ00034968 PinICL 13/10/1999
FUJ00032246 = PinICL 12/11/1999
FUJ00034278 = PinICL 15/11/1999
FUJ00034731 PinICL 15/11/1999
Extracted Text
"Call is currently under investigation by EDSC Team Member: Paul
Sausman""TIP may have observed the session 72134 did not balance.
Please arrange reconciliation then return for investigation into the
underlying fault.""There have been several calls where we in SSC
believe the call can be closed but TIP have refused to agree closure
"because we are approaching a crucial date".""I have spoken to Ian
Senior about closure but he refuses to agree as it is the subject of an
acceptance incident.”
"tip reconciliation missing transactions. live trial:miss match for fad
code 181329 cash account week 19 for cash account line 2050 as
follows: pathway derived =£36258.48, and tip derived = £34808.15: a
difference of £1450.33. this indicates that a transaction totalling
£1450.33 has not been passed to tip. this anomolly is repeated for the
following offices - 261329 (line 2051£44360.00), 310329 (line
2050£3750.25), 402329 (line 2051 £6533.90) and office 209511 (line
2050 £40.00).""Incident under investigation""I have rasied RED 527 to
infirm POCL that this is being invetigated. Please find out what the
differences are and the reason fior the mismatch.""All missing
transactions are related to Al 376. Can we please estanlish what the
missing transactions were? What was the cause of the problem?”
"a comparison between values received within cash account files, and
those derived from the transaction stream have id’d the following
anamolies...""This has the potential to cause us to fail to meet the
‘AI376 rectification plan and need urgent resolution.”"This change is to
reduce possible occurrence under Acceptance Incident 376."
"comparison values was made betwen the cash account file and those
derived from the transaction stream has identified a problem""PLEASE
NOTE THAT THIS COMES UNDER THE REMIT OF AI376. PLEASE
INVESTIGATE ASAP. THIS MAY NEED TO GO TO STEVE
WARWICK/PHIL HEMINGWAY (DEVELOPMENT) AFTERWARDS FOR
FURTHER COMMENT." This is the latest occurrence of this type of
misbalance that requires investigating also as part of this system call.
This occurred under pc33269.""This is an acceptance issue - please
deal with appropriately.""Investigation of the issue at these two offices
indicates that the problem lies with transactions recorded at each
office against product 2289 in Recovery Mode. The Pathway system
has (correctly) mapped these transactions to the AP line (0009) of the
Cash Account but TIP have assumed that these transactions should
map to the Local Products line (0059). This has been discussed and
confirmed with Dave Salt of POCL TIP Project.”
"THIS CALL NEEDS INVESTIGATION BY SSC, THEN IT MAY NEEDD TO
GO TO STEVE WARWICK (DEVELOPMENT) FOR FURTHER INPUT. THIS
IS COVERED BY AI376."" I can find no explanation for why TIP have
calculated a value different to that reported on the Cash Account”
Issue attempted to be closed but then: "This incident has NOT been
resolved. Steve Warwick said they only explanation we could see was
that the transactions were either not sent to TIP or were not
accounted for correctly by them, and TIP's response is that they both
received them and correctly accounted for them."
“comparison was made between the values recieved within the cash
acc files and those derived from the trans stream"" THIS CALL NEEDS
INVESTIGATION BY SSC, THEN IT MAY NEEDD TO GO TO STEVE
128
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
URN Ticket Date Extracted Text
Source
WARWICK (DEVELOPMENT) FOR FURTHER INPUT. THIS IS COVERED
BY AI376""l can find no explanation for why TIP have calculated a
value different to that reported on the Cash Account.”"The
transactions recorded at the counter are entirely consistent with the
Cash Account data recorded at the counter.""Can SSC please check to
see whether the 2 transactions at FAD 183306, as mentioned in John's
comments, were correctly sent to TIP.""As was explained to Nicole,
David Salt (POCL-TIP) supplied the detailed info about the 2
transactions (08/11/1999 13:24:03 3.00 and 10/11/1999 14:28:40
441.40) which was used in John Pope's comment. Therefore, TIP has
received the 2 transactions. Routing call back to Nicole
Meredith.""POCL have now been updated on the above responses via
RED 1355. Awaiting confirmation of closure." Ticket then closed with
no clear resolution. "This incident has NOT been resolved. Steve
Warwick said they only explanation we could see was that the
transactions were either not sent to TIP or were not accounted for
correctly by them, and TIP's response is that they both received them
and correctly accounted for them."
FUJ00036136 PinICL 04/12/1999 "A misbalance to stock unit was caused by cheque settlement of P&A
transactions. When balancing the stock unit, a warning message that
receipts did not equal payments was not ouptut. The message did
appear for the Trial Cash account.""It is acutely embarrassing that this
has stopped working - that it should work is a specific contractual
requirement.”
FUJ00036863 PinICL 09/12/1999 "This call is related to AI376 and will require resolution before the
recommencement of Rollout in January.""Acceptance Incident:
Al0376H"Problem in a Scales transaction"'The APS transactions are
all occurring in recovery mode and the APS team has been asked to
look into the problem."
FUJ00034224 PinICL 13/12/1999 "When I looked in the message store for £4.16 or -£4.16 I found
17.1.12
under <Id:2> an instance of selling product260 and its settlement -
looks innocuous.""The problem at outlet 8323 appears to have
originated in CAP 35 and was caused by a transfer of stock between
two stock units for a total value Of £428.10. The transfer appears to
have caused an imbalance in the office during CAP 35 with the total
Payments being greater than total Receipts by £528.20 (twice the
value of the transfer). This therefore meant that the Balance Due to
Post Office (line 1085) on the CAP 35 Cash Account was £528.20
higher than it should have been, causing the Balance Brought Forward
(line 0001) on the CAP 36 Cash Account to be similarly affected. The
stock lines on table 5 of the CAP 35 and Cap 36 Cash Accounts were
similarly affected.""The cause of the imbalance in CAP 35 at 008323
was that a ‘Session Swap’ was made between nodes 7 and 1 while the
user was in the middle of the Transfer In.""The discrepancy of £4.16
at 322420 arose from the reversal of an APS transaction.""Acceptance
Incident : AI0376H"
I surveyed the PPs for the pattern of “Acceptance Incident” followed by a numeric or “AI”
followed by a numeric to identify PPs that dealt with Acceptance Incident issues. The
following figure shows that 358 PPs were related to Acceptance Incidents and 223 PPs were
specifically associated with AI 376 (accounting integrity). The pattern on this figure follows
the narrative derived from the monthly reports.
129
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 17.1 Monthly references to AIs in the PPS
Acceptance Incidents
80
70
60
50
40
30
20
* hi
o- ft l] ne ee
SRE ee SHN BARE HERE BeSANEBA
SS eegnaneesegsseeeesega tes 8
a ar or ae ar a ae er ae a a a a a a
RSRRSSERESSSESSSSSSRSS
RRRRRRRSSESSESEEEE EBB
SAGA FessegegSRCRCRSRRRRKRRR
All Als mAI 376
130
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
18. Payment and receipt imbalances were common symptoms
with varied causes
18.1 Background
18.1.1 The accounting integrity issue highlighted in the previous section directed me to identify
18.1.2
18.1.3
examples of accounting issues within the PPs. This section of the Report explores the
selected examples of accounting issues as represented by payment and receipt imbalance
issues.
Section 4.7 of this Report provided an overview of the balancing and roll-over process that
was used in LHITS. I have reproduced the example from that section for ease of reference,
but I have updated it to show how a payment and receipt imbalance could occur.
In this example a bug in the system causes the brought forward balance to be incorrectly
calculated as three times the correct value (later in this section are details of a PP
(PC0027139) where it is documented that a scenario akin to this occurred in LHITS). The
brought forward balance was calculated as £16,500, rather than the correct value of £5,500
(the red text shows where the error occurs).
Table 18.1 Receipts and Payments Account for AA in CAP15
Brought forward balance from CAP14 into CAP15 for AA:
Cash £5,000 £15,000
Stock £500 £1,500
Receipts (debits) Receipt amount Payments (credits) Payment amount
Payment for TV Licence £100
Payment of road tax £75
Alliance & Leicester Giro £150
deposit
Purchase of 20 x 1* class £5
stamps for cash
Additional money received £100
("remmed in”) from POCL
A&L Giro withdrawals £50
Pension payment £25
National Savings £100
withdrawals
Issue of 20 x 1* class £5
stamps to a customer
Carried forward balance from CAP15 to CAP16
for AA:
Cash £5,255
Stock £495
£57930 £16,930
131
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
18.1.4 This shows an imbalance of £11,000 (payments greater than receipts). If the LHITS error
was not known, this would appear as a shortfall of £11,000.
18.1.5 What is also apparent from this example is that there are various other issues that could
result in an imbalance, for example:
(a) There are payments that were not recorded in the LHITS, or payments that were
erroneously recorded in the LHITS;
(b) There are receipts that were not recorded in the LHITS, or receipts that were
erroneously recorded in LHITS;
(c) The carried forward balance was incorrect because the cash and/ or stock were not
correctly declared by the SPM, or there have been cash and stock changes that cannot
be accounted for.
Methodology
18.1.6 I Working with the PPs loaded into Brainspace, an initial search was run over the PPs to
identify those which contained the Concepts of: error, cash, issue, fail, and others as shown
in the figure below:
Figure 18.1 Brainspace concept search results
TORIC
18.1.7
18.1.8
ONCEPTS
2 error ? same i following error
me error me:
r number system error
fatal error : 5 B : fail
failure
This search returned 38,803 PPs, however only a small number of these were weighted
highly relevant to each of the displayed concepts.
Reviewing those at the top of the distribution revealed an issue mentioned in several PPs,
namely cash or stock not balancing, with a common phrase used being “receipts vs
payments”. This was input into a new, more focused search, which included the following
additional concepts as being closely linked:
132
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
EXPG0000001
EXPG0000001
Figure 18.2 Brainspace concept search results
18.1.9
18.1.10
newly migrated
confrmed
time of migration
ly
It is worth noting that “EnteredBBF” is commonly found in the PPs when pasting in error
messages from a manual migration (“MiMan”) message store.
This search returned 67 documents, which were manually reviewed in order of descending
likely relevance. Once 11 documents were identified as relevant, a CMML model was created,
which identified a set of documents likely to also be relevant to the same concept.
18.1.11 Three additional rounds of training were undertaken, as set out in the table below:
Table 18.2 Verbatim extracts from Monthly Reports
Round Type Coded Docs in Comments
(T: P/N) 0.8-1.0
Range
1 Manual 22: 1,334 Example documents were provided to the model for the first round,
4a / it resulting in a high number of likely relevant documents and
additional refinements required.
2 Top 32: 576 The ten documents with the highest scores were reviewed,
Scores 16/16 resulting in five documents being coded positive, and five
documents coded negative.
3 Top 42: 955 The ten documents with the highest scores were reviewed. This
Scores 26/16 time all 10 documents were coded positive.
4 Diverse 51: 386 To further refine the model, we ran a “diverse active” round to
Active 28/23 sample a mixture of ranked documents. Two were coded positive,
18.1.13
and seven coded negative. The model was refined and resulted in
386 documents being ranked in the highest relevancy tier.
Due to the differences in how each system stores the rank number (Brainspace assigns a
cut-off from 0.8000 whereas Relativity rounds up from 0.7950), there were 399 documents
ranked 0.8 or higher (highly relevant to this concept). This is not necessarily all “relevant”
documents however it is the set which are most likely to be relevant based on the seed
documents provided to Brainspace.
Of the 399 PPs, 137 were selected for review by my team, which identified 127 PPs as
relevant to the concept of payment and receipt imbalance. The following section contains
an analysis of this population.
133
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Quantitative observations
18.1.14 We targeted two standard codes present in the PP text:
(a) Response Category: I understand that this denotes “the call status within the
lifecycle”®*, with different options available depending on the call type of the PP.
(b) Defect cause (note this term is used interchangeably with ‘root cause): I
understand this was captured for the purpose of supporting root cause analysis so as
to”... ensure the same errors do not occur twice.”. A defect cause value is mandatory
for all new calls and the defect cause may change during the life cycle of the call as
investigation matures and a better view of the problem is identified”°.
18.1.15 Since the ‘Response Category’ and ‘Defect cause’ are refined during the lifetime of a PP, I
selected the final chronological values for these analyses.
Analysis of ‘Response Category’
18.1.16 The figure below shows the final ‘Response Category’ extracted from the PPs.
‘igure 18.3 Response categories for the reviewed payment and receipt
imbalance PPs
Other
6%
Duplicate Call
6%
No fault in product
7%
Avoidance Action
Supplied Reconciliation
2% - resolved
Advice and guidance 38%
given
5%
Published Known
Error
5%
Fixed at Future
release
6%
S/W Fix Administrative
Released to Call Response
Logger 13%
Based on this data I make the following observation:
« PinICL Reference Data Guide, version 2.0 dated 18 February 2002, section 8 (FUJ00098258).
6 Submissions on behalf of Fujitsu Services Limited dated 13 September 2022 (in response to a Rule 9 Request dated
29 April 2022) (FUJ00119556)
7” PinICL Reference Data Guide, version 2.0 dated 18 February 2002, page 10 (FUJ00098258).
134
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
18.1.17 Whilst I do not have formal definitions for the values shown in the figure above, I conclude
that 19% of the closure reasons (Administrative Response, Other) do not provide much
insight into the investigation process.
Analysis of ‘Defect cause’
18.1.18 The figure below shows the final ‘Defect cause’ extracted from the PPs.
Figure 18.4 Defect causes for the reviewed payment and receipt imbalance PPs
General - User
Knowledge
5%
General - User
10%
Development -
Code
General - in 33%
Procedure
7%
Gen - Outside
Pathway Control
9%
Development -
Low Level Design
2%
Development -
Reference Data
Yo
General - fo
Design - High Level
Unknown Integration - Build ie
25% 1% ;
18.1.19 Based on this data I make the following observation:
(a) A significant proportion of these PPs had defect causes that were recognised as being
related to the design or development of LHITS (45%)’?. This indicates to me that
there were acknowledged bugs, errors, or defects in LHITS that were capable of
giving rise to a payment and receipt imbalances.
Days to resolve:
18.1.20 The figure below shows how long the reviewed PPs remained open.
” Whilst formal definitions of these defect codes were not available to me, I have assumed, prima facie, that the defects
of ‘design — high level design’, ‘Development - code’, ‘Development - low level design’, and ‘Development - reference
data’ all relate to acknowledged bugs, errors, or defects in the LHITS.
135
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 18.5 Time to resolve PPs
40
37
35
30
26
25 22
18
20 16
15
10
; I I
i)
1 week 2 weeks 3 weeks 4 weeks 5-8 weeks 9 weeks of
more
Time time resolve (in weeks)
18.1.21 Based on this data from the figure above, I make the following observations:
(a) Only 26 PPs (20%) were fully closed within a week.
(b) 55 of the PPs (43%) took 5 weeks or more to fully close.
(c) 37 of the PPs (29%) took 9 weeks of more to fully close.
136
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
18.2
Qualitative observations
18.2.1
18.2.2
18.2.3
18.2.4
18.2.5
PPs are technical documents: their interpretation is sometimes difficult. I have provided an
example PinICL and PEAK in Appendix C in order to fully illustrate the challenges of
interpreting these.
To illustrate the payment and receipt imbalances that occurred during the national rollout,
it is useful to examine individual PPs. Therefore, in this section I have selected seven PPs
that highlight the varied causes of the payment and receipt imbalances.
For the benefit of the reader, I have structured my review of the seven PPs into the following
form:
(a) Summary - High-level information relating to the PP
(b) Chronology - Selected excerpts from the PP’s comments
(c)__ My observations
Based on my review of these documents I make the following general observations:
(a) Many of these PPs seem to have been raised as a result of internal reconciliations.
(b) I There does appear to be an earnest effort, on the part of the SSC, to investigate
these issues, identify a root cause, and mitigate future recurrences.
(c) The tickets show that different teams were involved when investigating these issues.
(d) In the majority of these PPs, it is not evident that the identified issue was resolved.
(e) In a majority of these PPs, the root cause is related to LHITS.
The following tables contain verbatim extracts from the PinICLs and PEAKs (PPs). I have
intentionally not made any corrections to grammar or spelling. The views expressed in these
extracts are that of the authors.
137
EXPG0000001
EXPG0000001
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Table 18.3 (1) ECCO Migration - User error or ECCO system issue
Summary
Reference (PP PC0038771 / FUJ00037419
/ URN)
Format PinICL
Date opened 18 February 2000
Date closed 25 February 2000
Days openfor 8
Original call c
priority
Final Category Category 68 - Administrative Response
Final defect 40:General - User
cause
Chronology
Team (Member) Date Entered Extracted Comments
Customer Call 2000-02-18 this is a system call related to tip 1052. please route this call to
13:48:26 john moran in edse
18/02/00 13:39 uk080008
Advice: asked to reassign by lohn moran
MSU 2000-02-18 F} Response :
(ohn Moran) 14:31:03 SESS IIIS IIIS DIII IIA IAAI III IA IIIA AO
SRI AOI III III IO
The following has been copied from business call e-
0002180320/pc0038730 TIP incident 1052. Descrepency in Cash
Account for week 46 (Week ending 9/2/00) a comparison between
values recieved within the cash account files and
those derived from the transaction stream for FAD 0051136 ORG
unit 17831 identified the following differences:
Cash Account line 2050 declared amount 125780.59 derived
amount 125683.20 diff of 97.39, Cash account line 2051 declared
amount 0 derived amoutn 97.39 difference -97.39. Reasons are
required.
JBI OO IOI IO II IU IO I IO RII II I a a I
SBI GOI IODIDE
SSC Please investigate and attach message store. I suspect Steve
Warwick will want a look at this...
EDSC 2000-02-21 F} Response :
(Garrett Simpson) 10:19:31 This is an illustration of the stupidities that ECCO software allows.
A clerk can transfer cash and cheques between stock units without
bothering to make sure they match up. The result shows up when
the transaction data is migrated to Horizon which insists on clear
demarcation between cash and cheques. In this case the critical
transactions are Transfers In of two cheques whose total amount
exactly equals the discrepancy noted.
Details are attached in file DuffTrans.txt.
No fault in Horizon product.
MSU 2000-02-22 F} Response :
(John Moran) 09:05:47 I think Steve Warwick is aware of the ECCO problem here but as a
matter of course I will route the call to him to allow him to
comment...
QFP 2000-02-22 F} Response :
(Steve Warwick) 10:36:51 This issue is well documented in previous incidents with TIP. The
effect is that the Pathway system reports the values of the
affected products (in this case Cash and Cheques) incorrectly on
138
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
EDSC
(Garrett Simpson)
MSU
(John Moran)
Customer Call
the Cash Account for the migration CAP, although the Cash
Account still balances. TIP then use the Cash Account figures from
the migration CAP as the start point for validating the next Cash
Account received from the outlet and report a discrepancy
between the transactions received in week 2 and the Cash Account
for week 2. This is a user error pre-migration of an ECCO+ Office.
2000-02-22 F} Response :
16:13:12 Passing to MSU for issue of RED.
2000-02-25 _F} Response :
#is32:0 Final red 2078 issued to customer. Not data error. please close
this call.
2000-02-25 Date and time complete: 25/02/2000 11:45:21
11:48:57 Service Complete (Confirmation) Received
My observations
Was the
immediate
issue fixed?
Was a defect/
root cause
identified?
Was this
defect/ root
cause correctly
recorded in
the PP?
Is there
evidence that
this defect/
root cause was
addressed?
Observations
on the
management
and closure of
the issue.
Observations
‘on defect /
root cause
If I assume that the issuing of the RED 2078 notice to the customer resolved the PP, then
yes.
It appears from the text that this is a known issue with the ECCO software used in the
branch pre-Horizon and that the TIP reconciliation process had previously identified
instances of this.
The root cause is identified as ‘40:General - User’. Based on the text this would appear to
be correct.
As this is a user error, I would not necessarily expect the root cause to be addressed,
especially given that migration from ECCO to LHITS is a one-off event
It appears from the text that the origin of the PP was an internal reconciliation control, so
no response to an SPM was required.
The linked PinICL (pc0038730) was closed on 06 March 2000 with the final status of
‘Category 90 -Reconciliation - resolved’.
Based on the text of the PP I agree that a ‘40:General - User’ defect cause is correct.
139
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Table 18.4 (2) TIP-related issue with existing reversal transactions
Summary
Reference (PP /
URN)
Format
Date opened
Date closed
Days open for
Original call priority
Final Category
Final defect cause
Chronology
Team (Member)
Customer Call
MsU
(Angela Shaw)
EDSC
(John Simpkins)
EDSC
(Jim Anscomb)
EPOSS
(Mark McGrath)
PC0028847 / FUJ00034029
PinICL
20 August 1999
30 December 1999
133
B
Category 60 - Fix Released to Call Logger
14:Development - Code
Date Entered
1999-08-20
14:18:50
1999-08-23
16:10:20
1999-08-23
16:44:59
1999-08-24
10:33:50
1999-08-24
11:18:27
Extracted Comments
Incorrect CA value. Live trial, the CA sub file for org
units 12609 (FAD
316523) CA week 21 contains an entry for line 2050
with a value of
£17181.05. However, TIP has calculated from the
transactions it has received
that the value of the line should be £17642.31. This
leaves a difference of
£461.26.
20/08/99 15:12 UKO61354
F} Response :
Barbara, I have just spoken to John Pope
(Requirements) this is classified
unde r Acceptance Incident 376 (AI). Would you please
raise the level to an
A/ Al incident. Would John Simpkins please take a
look, then send to EPOSS
Dev. Thanks
T have checked the agent boxes at wigan for any
T_HV_ALL event for this
office between 12-Aug-1999 and 18-Aug-1999 and did
not find any.
F} Response :
There is a null transaction Mode on -1-117305
- <Mode:> for a cash credit of gbp 143.22, though this
is now not a problem
for the harvester.
No delays shown in the APR db.
Send to EPOSS-dev
The erroneous message was 117938 not 117305 - in
case any one else is relying on this info.
We released a fix for this 20/8/99 into WP 5406 which
went to OTT and is due to be released in Tivoli package
EPOSS_COUNTER_CORE version 3_3. Thus, it
has not made it to live yet.
The problem message is unfortunately an Exisitng
Reversal messsage so the harvesters automatic
assignment to Serve Customer is likely to provide
140
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
EDSC
(Jim Anscomb)
EPOSS
(ohn McLean)
Unknown
(John Pope)
QFP
(Steve Warwick)
QFP
(Steve Warwick)
1999-08-24
12:32:24
1999-08-24
14:11:46
1999-08-24
18:31:07
1999-08-25
18:01:56
1999-09-07
15:58:10
EXPG0000001
EXPG0000001
problems, some one will need to amend this.
Routing to EDSC for them to solve the procedural
problems. - and check when the Tivoli package is due
for release.
Austin
F} Response :
The total discrepancy is for GBP 461.26, 143.22 has
been accounted for above
- can someone assist with any of the remaining 318.04.
THIS CALL IS ASSOCIATED WITH HIGH PRIORITY
ACCEPTANCE INCIDENT 376.
PLEASE PROGRESS RAPIDLY
Just a thought, but the sign reversal mentioned above
(serve customer setn to TIP instead of Existing
Reversal)may explain 2 X 143.22 = 286.44
Can anybody help with £174.82 ?
F} Response :
It may be of interest that the value of the discrepancy
between the TIP and Pathway figures appears to
correspond to 2 x £230.63. During the balancing of
stock unit AA on 18.8.99, a stock adjustment was made
to reduce the value of Cheques (Product 2) by this
amount, with a corresponding increase in Cash. These
two stock adjustment records were later individually
reversed, generating a further 4 transactions for
£230.63, 3 against Cash (Product 1) and 1 against
Cheques (Product 2). Therefore in total 4 Cash
transactions (two positive, two negative) and two
Cheques transactions (one positive and one negative)
were written.
Given that there have previously been issues with TIP's
rejections of 'Existing Reversal’ transactions where the
reversal settlement contained no cross-reference details,
is it possible that this has caused the reconciliation
failure? According to the message store data, the Cash
Account for CAP 21 reported Total Receipts = Total
Payments, indicating that the message store data is
complete and accurate.
The response has been flagged to the gateway team for
validation
F} Response :
From further information received from TIP, the
sequence of events seems to be as follows:
1. At 17:21:20 on 18.8.99 a stock adjustment was
carried out to reduce the value of cheques by
£230.63. This wrote two transactions - one to
reduce the value of cheques (17:21:20), one to
increase the value of cash (17:21:20) by the same
amount, both transactions carried the mode 'SAN’
(TIP - 18).
2. At 18:22:27 on 18.8.99 a reversal of THE CASH
SETTLEMENT transaction for the Cheque adjustment
took place resulting in two transactions being
written against Cash, one to reduce the value of
cash (18:22:27) and one to increase the value of
cash to settle the reversal (18:22:49), both
141
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
EDSC 1999-09-15
(im Anscomb) 13:48:14
EPOSS 1999-09-16
(Mark McGrath) 16:18:33
EPOSS 1999-09-16
(Mark McGrath) 16:34:22
Unknown 1999-09-17
(Gurdeep Atwal) 11:57:03
transactions carried the mode 'ER' (TIP - 1 with
reversal indicator).
3. At 18:24:32 on 18.8.99 a reversal of the CHEQUE
ADJUSTMENT transaction (see 1 above) was carried
out, generating two transactions - one to increase
the value of cheques (18:24:32) and one to reduce
the value of cash by the same amount (18:24:37).
These transactions are recorded in the message store
with the correct signs. From the information supplied by
TIP it seems as though they have received/treated the
transaction at 18:24:32 (a reversal of a previous
reduction in the value of cheques) as though it was a
reduction in value rather than an increase in value,
therby calculating a discrepancy of twice the amount.
Either the sign on the transaction value sent to TIP was
incorrect, or TIP have misinterpreted the data sent.
F} Response :
Looking at the tip file there were 2 reversals for 230.63
in quick succession, the first is translated for tip as
balancing + and - entries, the second however is
translated into two + entries, which would account for
the error. See extract of tip file and message store
attached.
Changes to be made to clsEPOSS and clsTransaction in
EPOSSCore. Fix applied to EPOSSCore. You should get in
the attribute grammar for a cash settlement for an ER
transaction the additional data of
CrossReference.Omode: <what ever the original mode
was>
The harvesters need this.
.-Austin
testing of this should include transacting in each mode:
the messages shoul
dbe as they were.
Then performing a reversal of each mode and checking
that the new attribute
grammar exists in the cash settlements ofthe reversals.
. Austin
Link tested OK on CSR dev counter (WP 5767)
Performed a tranaction followed by a exisiting reversal
for each of the following modes :
Serve customer, Rems (all modes), reval up/down,
House keeping, non-acc data, parcel traffic, bulk input.
On each exsisting reversal the message store was
checked for the new attribute grammer.
CrossReference.OMode - Followed by the corresponding
mode of the reversal.
EXPG0000001
EXPG0000001
142
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Live Supp.Test
(Nicola Lambert)
EDSC
(Garrett Simpson)
SSC Holding
(Catherine Obeng)
My observations
Was the immediate
issue fixed?
Was a defect/ root
cause identified?
Was this defect/
root cause correctly
recorded in the PP?
Is there evidence
that this defect/
root cause was
addressed?
Observations on the
management and
closure of the issue.
Observations on
defect / root cause
1999-10-26 WP_5766 has been applied to live. Routing call back to
14:39:35 call logger for
closure.
1999-10-27 F} Response :
09:17:51 We have seen that when a call is the subject of an
acceptance incident (as this call is) then there is no
point in us ringing the originator to ask for closure. They
always say that such calls are the subject of regular
discussions between John Pope at FELO1 and Martin Box
of TIP. Eventually somebody at TIP rings us with a list of
calls which can be closed.
Accordingly I shall send this call to our holding stack to
await such closure.
1999-12-30 F} Response :
14:19:02 Call closure agreed by call raiser, David Salt.
Yes, on the basis that the call raiser (David Salt) agreed to close the call on 30
December 1999.
A clear root cause was identified, being an issue with the data delivered to TIP.
A defect cause of “14: Development - Code” would seem consistent with the root
cause identified in the text.
The text indicates that a software update was made and tested, albeit the text
only indicates that the testing occurred, not the results of this. However, I note
that the text does not indicate that the tests failed, which I would expect it to
say had this been the case. The text also indicates that the software fix was
implemented.
The closure of the PinICL once the fix had been implemented and the original
raiser’s approval was obtained seems appropriate to me.
There is evidence in the ticket that a fix was implemented in LHITS to remediate
the identified issue.
143
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Table 18.5 (3) Reference data delivery issue
Summary
Reference (PP / URN)
Format
Date opened
Date closed
Days open for
Original call priority
Final Category
Final defect / root cause
Chronology
Team (Member)
Customer Call
MSU.
(John Moran)
MSU.
(John Moran)
EDSC
(Paul Steed)
Customer Call
PC0051382 / FUJO0064777
PEAK
28 July 2000
01 August 2000
5
B
Category 90 -Reconciliation - resolved
99: General - Unknown
Date Entered
2000-07-28 15:52:22
2000-07-31 12:53:40
2000-08-01 13:23:45
2000-08-01 14:05:02
2000-08-01 14:10:32
Extracted Comments
28/07/00 16:48 office 91008 reports a difference in
its reciept & payment totals for cap18 . please send
this call to john moran
I know what caused this problem. It was because
reference data was not sent to the outlests
concerning P&A products--The cash settlement was
mapped to the CA, but the corresponding transaction
was not. If these transactions were recorded by in
the Counter Transaction Exceptions report I could
supply POCL TP with this information myself, but
they have not been recorded.
Can you supply the offending non mapped
transactions to this PinICL in message store extact
so I can reconcile with Chesterfield?
F} Response :
This difference in the receipt and Payment totals was
caused by the fact that non-core reference data was
not delivered to this office in time. The reference data
was for OBCS products 177 to 185. As this reference
data included primary mappings for these products
these products could not be mapped to the cash
account at stock unit rollover. This is what caused the
difference in the receipt and payment totals.
SO IOI
This incident is related to 9 others all caused by this
same problem. Alll the offices effected were migrated
to Horizon on 20/7/00. All the offending transactions
took place 21/7/00-- when there was not reference
data at the outlets. The correct reference data was
delivered for business on 22/7/00.
SEBO OIA
I have provided with the final BIM report an excel
spread sheet (with the same file name as the BIM
report) listing the offending transactions which were
not mapped to the cash account.
F} Response
Caller has raised the BIM based on the evidence
extracted and so call can be closed (Reconciliation
Resolved).
Date and time complete: 01/08/2000 15:07:08
Service Complete (Confirmation) Received
144
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
My observations
Was the immediate issue
fixed?
Was a defect/ root cause
identified?
Was this defect/ root
cause correctly recorded
in the PP?
Is there evidence that
this defect/ root cause
was addressed?
Observations on the
management and closure
of the issue.
Observations on defect /
root cause
Whilst the text states that a BIM was raised and the impacted transactions were
identified and included with the BIM, it is not possible to determine, from the PP,
whether the transactions were updated to map them correctly. If I assume that
the BIM process would rectify this issue, then it appears that the appropriate
steps were taken to rectify the issue.
The text indicates that this was a known issue and a clear root cause is provided
(i.e., the product reference data required to correctly map the transactions to
the cash account had not been delivered to the branches, so the transactions
were not mapped).
The root cause recorded in the PP is “General - Unknown”. The root cause is
clearly defined in the PP and therefore a root cause more akin to ‘Product
reference data not delivered in time’ would seem like a more accurate and useful
root cause.
The text references the fact the branches had now received the required
reference data.
The specific issue was fixed in that the reference data was delivered to the
branch.
There is evidence in the ticket that a fix was implemented in LHITS to remediate
the identified issue. It is unclear to me why the reference data required was not
timely delivered to this branch, and nine others.
145
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Table 18.6 (4) Stock unit deletion
Summary
Reference (PP / URN)
Format
Date opened
Date closed
Days open for
Original call priority
Final Category
Final defect / root cause
Chronology
Team (Member)
Customer Call
EDSC
(Paul Sausman)
EDSC
(Paul Sausman)
EDSC
(Paul Sausman)
EDSC
(Paul Sausman)
EDSC
(Paul Sausman)
MsU
(Angela Shaw)
MSU
(Angela Shaw)
PC0028263 / FUJ00029840
PinICL
04 August 1999
29 September 1999
57
B
Category 60 - Fix Released to Call Logger
14:Development - Code
Date Entered
1999-08-04 11:31:23
1999-08-09 14:19:22
1999-08-09 15:03:15
1999-08-09 15:13:48
1999-08-09 15:29:40
1999-08-09 15:41:43
1999-08-10 17:07:52
1999-08-11 13:34:09
Extracted Comments
TIP- reconciliation - missing transactions live trial :
cash account week 18 office 230511, pathway
derived cash account line 2050 value = £36272.65 ,
TIP derived value = £36133.20, difference of
£139.45. this has a knock on affect to line 1085,
1700, 2072 and 2700. this is probably attributable to
missing transactions, although identical problems
were also identified at offices 013523 (£1936.38),
278523(£155), 101114(£15.41). PLS INVESTIGATE
04/08/99 12:26 UK061356
Information: Reconciliation issue - passing for
investigation.
These outlets do not appear to have been affected
by the harvesting issue of 28218 nor are they in the
spreadsheet of errant transactions.
Null modes (27321) appear to account for the
transactions lost from 230511:
-- AA, 26/07/99 12:36:19, product 1, quantity -1,
amount -46.20;
-- AA, 24/07/99 09:15:00, product 1, quantity -1,
amount -93.25.
A null mode (27321) appears to account for the
transaction lost from 101114:
-- AA, 23/07/99 18:08:53, product 1, quantity -1,
amount -15.41.
A null mode (27321) appears to account for the
transaction lost from :
-- AA, 26/07/99 11:16:59, product 1, quantity -1,
amount -1936.38.
Please arrange reconciliation then return for
investigation into 278523.
F} Response :
Paul, have raised RED 515 for this call. Would you
please send back to me when you have more info
and a reason why these transaction were not
included.
Thanks
Will these transactions ever get returned to TIP &
HAPS? Please update.
146
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
EDSC
(Paul Sausman)
EDSC
(Jim Anscomb)
MSU
(Angela Shaw)
Unknown
(John McLean)
EDSC
(Garrett Simpson)
QFP
(Steve Warwick)
EDSC
(Garrett Simpson)
MSU
(Angela Shaw)
QFP
(Steve Warwick)
1999-08-11 15:36:59
1999-08-23 13:47:03
1999-08-24 11:28:55
1999-08-24 14:11:03
1999-08-25 09:31:52
1999-08-26 09:02:37
1999-08-26 11:10:27
1999-08-26 15:20:08
1999-09-14 13:00:27
EXPG0000001
EXPG0000001
Transactions not sent because mode attribute was
null. They will not be sent
by the system to TIP or HAPS.
F} Response :
Cash Account week 18 was the first week for FAD
278523 - small discrepancies are acceptable during
that week.
F} Response :
Paul orginally provided me with the missing
transaction details for 3 of the 4 fads listed. I still
need the missing transaction details for 278523, as I
have to send the details to POCL for reconciliation
purposes. Is it not possible to resend the
transactions to TIP in this case? Can we progress
these missing transactions asap, as they come under
AI 376. please route back to MSU afterwards.
Thanks
THIS CALL IS ASSOCIATED WITH HIGH PRIORITY
ACCEPTANCE INCIDENT 376.
PLEASE PROGRESS RAPIDLY.
I have checked this message store but can find no
reason for the problem complained of in 278523.
Passing to development for further investigation.
F} Response :
The £155 error reported by TIP at FAD Code 278523
is almost certainly related to an MVL transaction
(Product 125 or 128). A number of these
transactions took place in the week and there was
also a Loss declared the previous week for this value
against Cash. The value of £155 was also
transferred between two stock units during the week
and a gain of £155 was recorded when balancing at
the end of CAP 18 (offsetting the Loss of £155
declared at the end of CAP 17).
Since there was no failure of the office to balance its
Cash Account, it would seem that either one of these
transactions has not been sent to TIP or TIP have
miscalculated the value of the transactions reporting
to the Cash Account.
F} Response :
We have now had an explanation from development
for the final office in this call. Passing to
management support for reconciliation.
F} Response :
Can SSC re-check for the last FAD details as per
Steve Warwick's last update. I am also requesting
that TIP re-investigate their findings too, as this is
due to the possibility of the above 2 scenarios.
Please re-send back to MSU.
Thanks
F} Response :
The cause of the imbalance at FAD Code 278523 was
the deletion of Stock Unit ZZ on 29th July 1999
before the EOD marker for the outlet had been
147
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
EDSC
(Garrett Simpson)
EDSC
(Garrett Simpson)
QFP
(Steve Warwick)
Unknown
(Sampath Kumar)
Unknown
(Sampath Kumar)
Customer Call
My observations
Was the immediate issue
fixed?
1999-09-16 10:51:32
1999-09-16 11:03:07
1999-09-21 18:01:52
1999-09-28 16:55:06
1999-09-29 13:30:50
1999-09-29 13:34:17
EXPG0000001
EXPG0000001
written. This meant that transactions carried out on
the stock unit totalling £155.00 (Declaration
Discrepancy) in CAP 19 were not reported to TIP.
I have told the originator that the cause of this
problem was the deletion of the relevant stock unit
and asked him to agree closure. He wanted to know
if there is going to be a change in the software to
prevent such deletion and when it is going to arrive.
It sounded as though he would agree closure once
he has a date.
This is not relevant to this call: we have explained
the discrepancy in the figures which is what, strictly,
is required.
F} Response :
David Salt rang back. I had spoken to Steve Warwick
in the meantime and found that the fix to prevent
such stock unit deletion went out to the live estate
on 15-Aug - so it has been in place now for a month.
David wants to wait until the end of September
before agreeing a close.
F} Response :
The original response given to TIP was based on the
fact that the symptom of the call appeared to be
similar to other calls which had been identified as
being signing problems. This initial view was.
provided along with the statement that the incident
was still under investigation and that once the
evidence had been examined the root cause would
be setermined. The root cause has now been
determined and John Pope has updated the
spreadsheet shared with POCL re. AI376. Closure
will be agreed between John Pope and Calum Craig
(POCL).
Passing the call to John Pope for confirmation of the
above.
F} Response :
Fix applied to the live system. David Salt (POCL TIP -
customer) agrees to close the call. Waiting to discuss
closure with John Pope, before closing call.
F} Response :
Fix applied to the live system. David Salt (POCL TIP -
customer) agrees to close the call.
Date and time complete: 29/09/1999 14:32:17
Service Complete (Confirmation) Received
FADs 230511, 13523 and 101114:
If I assume that the issuing of the RED 515 notice to the customer resolved the
PP, then yes.
FAD 27852
From my reading of the text, I do not see clear evidence that the deleted Stock
Unit ZZ issue being fixed for this FAD. However, David Salt agreed to close the
issue.
148
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Was a defect/ root cause
identified?
Was this defect/ root
cause correctly recorded
in the PP?
Is there evidence that
this defect/ root cause
was addressed?
Observations on the
management and closure
of the issue.
Observations on defect /
root cause
FADs 230511, 13523 and 101114:
The root cause was identified as mode attributes on the transactions had ‘null’
values, which resulted in the transactions not being sent to TIP.
FAD 278523:
The root cause was identified as a stock unit ZZ deletion before the EOD marker
had been written, which resulted in transactions not being sent to TIP.
The defect cause recorded was "14: Development - Code”. I assume that this
applied solely to FAD 278523. If this assumption is correct, this seems like an
appropriate defect code for FAD 278523. It is unclear whether this also applied
to FADs 230511, 13523 and 101114 which were a different issue.
FADs 230511, 13523 and 101114:
There is no discussion in the PP related to any action being taken to investigate
why the mode attribute values were “null.”
FAD 278523:
The root cause for FAD 278523 was addressed through the fix rolled out on 15
August.
The PP is somewhat difficult for me to follow as two distinct problems are being
discussed on a single PP. Regardless, it does appear that the SSC properly
closed the two problems discussed.
FADs 230511, 13523 and 101114:
This PP contains no information regarding “null” values present in the mode
data.
FAD 278523:
The root cause is clearly identified and based on the text of the PP, would be
prevented in the future via an update to LHITS.
149
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Table 18.7 (5) Brought forward balance multiplied causing a payments and
receipts imbalance.
Summary
Reference (PP / URN)
Format
Date opened
Date closed
Days open for
Original call priority
Final Category
Final defect / root cause
Chronology
Team (Member)
Customer Call
Customer Call
BusinessSupprt
(Nicole Meredith)
Unknown
(Barbara Longley)
EDSC
(Diane Rowe)
Unknown
(Barbara Longley)
QFP
(Steve Warwick)
PC0027139 / FUJ00027630
PinICL
24 June 1999
06 July 1999
11
A
Category 68 - Administrative Response
14:Development - Code
Date Entered
1999-06-24
15:22:00
1999-06-24
15:28:30
1999-06-24
15:57:03
1999-06-24
15:58:47
1999-06-24
16:40:35
1999-06-25
14:15:32
1999-06-28
08:07:40
Extracted Comments
cross refered to e-9906230224 , the receipts and
payments table's do not match at office 176328
when rolling over the cash account, even though the
bought forward figure is correct - this call needs to
be sent to ssc to attatch the messagedoor extract for
this post office, and then to development for
investigation24/06/99 16:16 UK061815
Information: paged pathway duty manager and
voiced smci duty team leader (Chris Gulliver)
regarding this call
HSH1 Information: Voiced Julia Bowes regarding
this call.
24/06/99 16:21 uk061537 HSH1 Information: If
this problem is not resolved in a couple of hours,
please contact Julia Bowes, Duty Manager, and
inform her.
We need to know the exact cause of this incident
and find out whether it
should have been fixed already.
F} Response :
Nicole Meredith has returned call to Diane Rowe
(EDSC) as she needs to know
the exact cause of this incident and find out whether
it should have been
fixed already.
The receipts and payments do not match. The
brought forward figure appears to
be correct. The details of the figures are on pc27105.
Nicole needs this
investigating. I have voice promted Steve Warwick. I
have attached the
complete message store.
F} Response :
Have spoken to Steve Warwick in QFP and he is
curently investigating the call.
F} Response :
150
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
EXPG0000001
EXPG0000001
Expert Witness Report of Charles Cipione, dated 14 September 2022
QFP
(Steve Warwick)
Unknown
(Lionel Higman)
EDSC
(Paul Steed)
BusinessSupprt
(Nicole Meredith)
1999-06-28
11:46:39
1999-06-28
12:22:24
1999-06-28
12:52:18
1999-06-28
13:14:03
Initial investigations have shown that the problem
arose at the time that the Office Trial Balance report
was produced. On the Office Trial Balance report
the brought forward value was £71k instead of £14k.
This appears to have been caused by the creation of
a correctional stock unit (Stock Unit 22) which was
additional to the normal stock unit (AA). Due to an
error in the code, when the stock unit balance
records are read the first stock unit (22 - first in
alphabetical sequence) is correctly identified as
having no Brought Forward! value from the previous
week, the system then incorrectly assumes that this
must be the migration week and generates a
brought forward value for the stock unit which is
incorrect. This error is being investigated for urgent
correction.
Further investigation of the discrepancy on the Cash
Account is continuing to make sure that this is the
only issue at the root of the problem.
F} Response :
Investigation of the Cash Account Receipts/Payments
mismatch shows:
1. The CA Snapshot was prepared (but not printed)
on 23.6.99
2. The CA Trial Report was prepared and printed on
24.6.99
The records generated fro the Trial print on 24.6.99
did not include the Remittance totals (In, Out or to
CHEC), giving incorrect Receipts and Payments totals
and a mis-match of £5709.01 (Payments lower than
Receipts). This appears to be an error in the CA
preparation process since the same set of records
prepared for the CA Snapshot the day before DID
include the remittance records.
Re-running the message store data on our
development system did not replicate the problem
and the Cash Account was correctly produced in a
balanced state.
Investigation of how this occured will continue,
however in the meantime if the office has not yet
rolled into CAP 14 then I suggest that they re-run
the CA Snapshot, CA Trial and Rollover the office.
On the evidence we have seen, I would expect this
to produce a correctly balanced Cash Account.
QFP decision no further LT1 fixes will be produced
between now and LT2 delta application.
Resetting target release to CSR.
I have spoken to Nicole and she is going to check
with the PO that the office has rolled over (or is
rolled over if it hasn't).
The PM has confirmed that the office has now been
rolled over to CAP14, so the CA snapshot and CA
trial cannot be re-run for the previous CAP.
151
Post Office Horizon IT Inquiry
EXPG0000001
EXPG0000001
Expert Witness Report of Charles Cipione, dated 14 September 2022
QFP
(Steve Warwick)
QFP
(Steve Warwick)
EDSC
(Paul Steed)
Unknown
(Barbara Longley)
BusinessSupprt
(Nicole Meredith)
QFP
(Steve Warwick)
1999-06-28
14:26:16
1999-06-28
17:13:36
1999-06-29
13:45:42
1999-06-29
15:21:14
1999-06-30
12:35:39
1999-07-01
08:18:28
F} Response :
Please confirm whether the Receipts and Payments
totals matched when the Final Cash Account was
produced (this can be determined from the
messages in the message store - look for attributes
<CashAccLine:700> and
<CashAccLine:1700>)
F} Response
The doubling (or multiplying) of the brought forward
value on the Cash Account has been traced to an
error in the changes which were delivered to correct
the previous problem related to the previewing of
the 'Final’ Cash Account.
THIS WORK-AROUND NEEDS TO BE BROUGHT TO
THE ATTENTION OF THE NBSC AND HSH HELPDESKS.
AS A MATTER OF URGENCY IN ORDER TO AVOID
CASH ACCOUNT IMBALANCES DURING THE NEXT
TWO CASH ACCOUNT PERIODS.
Part of the changes allowed the user to re-start the
production of the CA Snapshot, Trial or Final if the
process was interupted by returning to the menu.
Previously, if the process was interupted then the
user was required to re-run the Office Balance Trial,
Final, CA Snapshot, CA Trial and then the CA Final
(Rollover). Due to an error, if the user does not run
the CA Snapshot process followed immediately by
the CA Trial and Final reports then the system writes
a further 'Brought Forward’ transaction record each
time the process is interupted and re-started. This
causes the Cash Account Brought Forward value to
be multiplied up as many times as the process is re-
entered. This problem does not occur in the LT2
software (due for release on 10.7.99) due to the
restructuring of the Cash Account production process
in line with the recent CRs. Tests have been
conducted to demonstrate that LT2 does not exhibit
the same behaviour. In the meantime, for the
remaining two Cash Account Periods on LT1, the
work-around is to re-run the Office Balance Trial and
Final reports, re-run the CA Snapshot process and
follow this immediately with the CA Trial and Final
prints.
F} Response :
Forwarding to Nicole with information provided by
Steve Warwick.
F} Response :
Nicole Meredith has requested that this call be
downgraded to 'B' priority.
F} Response :
Is there still a problem with the creation of a
correctional stock unit? If so, then this needs to be
fixed for LT2. Please pass to Development for the
attention of Steve Warwick.
F} Response :
The issue with the correctional stock unit has been
tested on the LT2 software and the situation is
handled correctly, no imbalance is caused.
152
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
EDSC
(Paul Steed)
BusinessSupprt
(Nicole Meredith)
EDSC
(Paul Steed)
Customer Call
My observations
Was the immediate issue
fixed?
Was a defect/ root cause
identified?
Was this defect/ root
cause correctly recorded
in the PP?
Is there evidence that
this defect/ root cause
was addressed?
Observations on the
management and closure
of the issue.
Observations on
defect/root cause
1999-07-01 F} Response :
14:56:43 Nicole, Steve has added comments as requested.
Can you please agree closure.
1999-07-06 F} Response :
08:13:06 I agree closure of this call, on the basis that the
problem will not re-occur in LT2.
1999-07-06 F} Response :
08:35:24 Closing call following Nicole Meredith's agreement.
Closure Code:Software Error
Repair Code:Fixed in Next Release
1999-07-06 Date and time complete: 06/07/1999 09:39:18
08:39:35 Service Complete (Confirmation) Received
I do not see evidence in the PP that suggests that the issue at FAD was fixed,
although the text does note that the FAD completed their roll-over, presumably
with the imbalance still in place. Perhaps the alert to the Horizon Helpdesk
indicates that the corrective procedures were communicated and executed.
Yes, the root cause was an unforeseen consequence of the current LT1 fix.
The defect cause recorded was “14:Development - Code”. This seems like an
appropriate code as software fixes were required to address the root cause.
Yes, a correction was present in LT2 that corrected this issue.
The root cause was identified, addressed in the upcoming LT2 release, and a
workaround was communicated to the helpdesk.
The root cause was identified and addressed in release LT2.
153
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Table 18.8 (6) Navigating to a different ‘mode’ while transactions are on the
stack
Summary
Reference (PP / URN)
Format
Date opened
Date closed
Days open for
Original call priority
Final Category
Final defect / root cause
Chronology
Team (Member)
‘Customer Call
MSU
(Angela Shaw)
MSU
(Angela Shaw)
EDSC
(Richard Coleman)
EDSC
(Lina Kiang)
Unknown
(Barbara Longley)
MsU
(Angela Shaw)
QFP
(Steve Warwick)
PC0032855 / FUJO0039260
PinICL
05 November 1999
23 March 2000
140
B
Category 90 - Reconciliation - resolved
16:Development - Reference Data
Date Entered
1999-11-05
12:27:41
1999-11-05
13:29:29
1999-11-05
13:45:43
1999-11-09
09:57:13
1999-11-10
11:10:11
1999-11-26
11:34:58
1999-12-15
15:15:28
2000-01-11
14:53:13
Extracted Comments
05/11/99 12:08 There has been a receipts and
payments misbalance in CAP 31 where 28 offices in
the first CA week after migration had this problem.
Please investigate why this has happened. Evidence
will be sent to SSC.
F} Response :
this call is the system incident for pc32811. route
back to msu for closure afterwards.
PLEASE NOTE THAT THIS NEED PROGRESSING
RAPIDLY UNDER AI376. THANKS
F} Response :
PRESCAN: Possibly due to errors accepted at
migration
F} Response :
As suspected, 24 of the 28 FADs had their
differences accepted at migration. The remaining 4
FADs (097136, 265420, 006434 and 249715) should
be investigated by EPOSSDev, however the following
was noticed and Dev should determine if relevant
and what it means:
263420 Table 3 UNCHARGED Receipts: Migration of
15.20 249715 <Application:MiMAN>
<Table:Table3><Prod:2654><Value:18.28>
Routing to EPOSSDev along with 4 message stores
as evidence.
F} Response :
Acceptance Incident p MSU would like this to be
progressed quickly.
Can the remaining 4 offices be investigated and
returned to MSU with update. Thanks
F} Response :
Having discussed this issue with Roger Donato and
the EPOSS Development team it is clear that this call
is a dupliacte of PCO035507. The code which
retrieves transactions at the end of day, creates both
the daily transaction count and the daily cash
account table totals. Therefore any transaction
omitted from the daily count will also be omitted
from the daily CA Table totals.
154
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
EXPG0000001
EXPG0000001
Expert Witness Report of Charles Cipione, dated 14 September 2022
QFP
(Steve Warwick)
FP
(Steve Warwick)
QFP
(Steve Warwick)
QFP
(Steve Warwick)
QP
(Steve Warwick)
MSU
(Angela Shaw)
MSU
(John Moran)
Customer Call
2000-01-11
14:54:20
2000-01-11
15:14:57
2000-01-11
16:34:42
2000-01-11
16:42:09
2000-01-11
16:48:28
2000-03-15
12:06:56
2000-03-23
13:15:32
2000-03-23
14:24:44
F} Response :
Apologies, the last update was related to a different
call, please ignore.
F} Response :
At FAD Code 006434 a Housekeeping transaction
was carried out on 27.11.99 for a value of £400.00.
This transaction was not settled and the user
navigated to the Revaluation Up menu and carried
out a transaction to revalue stamps up by £5.88.
These two transactions were then settled against the
revaluation settlement product (which does not
accumulate to the balance). The Housekeeping
transaction for £400.00 should have been settled by
Cash. This error (allowing the user to navigate to a
different 'mode’ while transactions are on the stack)
has now been corrected in the Live software.
At FAD Code 097136 there was a recorded
discrepancy at migration of £19.46 (the RED report
indicates a Receipts <> Payments difference of
£18.46). Unless pressed by POCL, pursuing the
cause of the migration discrepancy being reduced by
£1.00 appears to be a pointless exercise. The cost
of investigation has already exceeded this value
many times.
F} Response :
At FAD Code 249715 a Housekeeping transaction at
07:48 on 6th October against product 2655 for
£18.28 was settled by the settlement product for
‘Revaluation Up'. This was probably caused by the
user navigating to the Revaluation menu after
adding the Housekeeping transaction to the stack
and then settling while in the Revaluation menu.
(See above for a similar scenario at 006434). This
navigation problem has already been addressed in
the software which is now live at CI2_2.
At FAD Code 265420, a Housekeeping transaction
was carried out on 27.11.99 for a value of £15.20.
This transaction was not settled and the user
navigated to the Revaluation Up menu and carried
out a transaction to revalue stamps up by £15.20.
These two transactions were then settled against the
revaluation settlement product (which does not
accumulate to the balance). The Housekeeping
transaction for £15.20 should have been settled by
Cash. This error (allowing the user to navigate to a
different 'mode' while transactions are on the stack)
has now been corrected in the Live software.
F} Response :
Final update sent to POCL on the 15/3/00. Awaiting
closure.
F} Response :
No longer of interest to POCL. This incident was not
included on the list of call to remain open. I received
this list at the monthly Incident Mgt review meeting
from Jacqui Cave. As it is not on the list please close
this call.
Date and time complete: 23/03/2000 14:20:45
Service Complete (Confirmation) Received
155
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
My observations
Was the immediate issue
fixed?
Was a defect/ root cause
identified?
Was this defect/ root
cause correctly recorded
in the PP?
Is there evidence that
this defect/ root cause
was addressed?
Observations on the
management and closure
of the issue.
Observations on defect /
root cause
For 25 of the 28 FADs the differences had been handled at the time of migration.
For the remaining 3, I do not see explicit evidence in the PP that suggests that
the issue with the FADs was fixed, but POCL’s exclusion these FADs from their
“list of call to remain open” seems to indicate that the issue was resolved to their
satisfaction.
The root cause for 25 of the 28 FADs related to issues in their initial migration.
For the remainder, the root cause was identified, namely that LHITS allowed a
user to navigate to a different mode while transactions were on the stack, which
would then be subsequently settled incorrectly.
The defect cause recorded was "16:Development - Reference Data”. Without a
comprehensive understanding of which component of the system contained the
error it is not possible for me to say whether this is an appropriate code.
The PP states that the error was corrected in the Live software.
This PP was closed without positive confirmation that the imbalances were
properly addressed.
The root cause appears to have been identified and remedied.
156
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Table 18.9 (7) DataServer tree build failure
Summary
Reference (PP /
URN)
Format
Date opened
Date closed
Days open for
Original call priority
Final Category
Final defect / root
cause
Chronology
Team (Member)
Customer Call
EDSC
(Diane Rowe)
EDSC
(Diane Rowe)
QFP
(Steve Warwick)
QFP
(Steve Warwick)
QFP
(Steve Warwick)
QFP
(Steve Warwick)
PC0045061 / FUJO0067416
PEAK
16 May 2000
14 September 2000
122
B
Category 90 - Reconciliation - resolved
41:General - in Procedure
Date Entered
2000-05-16
15:46:45
2000-05-19
0 55
2000-05-19
09:42:06
2000-05-23
17:09:58
2000-05-23
17:10:45
2000-05-24
10:32:04
2000-05-30
:57
Extracted Comments
THe host generated cash account line comparisons report dated
15/5 where post office 169207 has a difference in the recipts and
payments total for cap 06.Please investiagte”
F} Response :
This office did not have a migration discrepancy.
F} Response :
This office has had big problems with its receipts and payments.
Cap 5, 6 and 7 did not match. The differences are:
CAPS 16284.72
CAP6 -19296.15
CAP7 14526.08.
The office has already reported problems balancing which are
being investigated by development - see pc43811 (E-
0004271707). I have attached the complete messagestore.
F} Response :
This is a duplicate of PinICLs 43811 and 45061 which are already
under investigation
F} Response :
My apologies, this IS 45061!
F} Response :
"The cause of the problems in all three CAPs at this outlet was the
fact that Stock Unit DD's rollover records from CAP 5 to CAP 6
represented a ‘nil’ balance (the total stock holding was nil, no
receipts or payment transactions were recorded) despite the fact
that the stock unit had been trading normally during the period.
This issue was raised in PinICL 43811 and is still under
investigation within the EPOSS Development team.
The fact that Stock Unit DDs transactions and stock holdings were
omitted from the CAP 5 Cash Account meant that the Brought
Forward value for the Office in CAP 6 was incorrect. This caused
the CAP 6 Cash Account to misbalance. I am still investigating why
the CAP 7 Cash Account misbalanced, but I note that the office
returned to a balanced position in CAP 8.
F} Response :
30/5/00: On further investigation, the same problem that affected
stock unit DD in CAPS affected Stock Unit TT in CAP 6, i.e. at
balancing time the system failed to record the correct stock
157
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
EPOSS
(Les Ong)
EDSC
(Diane Rowe)
MSU
(John Moran)
FP
(Steve Warwick)
EPOSS
(David Linten)
EDSC
(Diane Rowe)
MSU
(ohn Moran)
EDSC
(Martin McConnell)
2000-07-04
17:57:23
2000-07-05
12:53:49
2000-07-05
16:09:30
2000-07-06
16:17:59
2000-07-10
13:26:19
2000-07-12
09:12:03
2000-07-12
09:47:52
2000-07-12
12:29:00
EXPG0000001
EXPG0000001
holding for the stock unit and failed to write the summary totals
for the Receipts and Payments products. The only records written
were the declared Cash and Stamp holdings with a discrepancy
equivalent to these amounts. This failure will have compounded
the CAP 6 problem with stock unit DD and then generated a
further discrepancy in CAP 7. I am passing the call to EPOSS-FP so
that the message store evidence of the problem in both these
CAPs can be examined.
F} Response :
This problem is the same as that already resolved on PinICLs
37884 & 37663, namely that of DataServer not tree building &
populating correctly. A diagnostic has been put into DataServer to
detect any such problems.
F} Response :
Please can we agree closure on this now? See previous updates for
details.
F} Response :
I thought diagnostic code was delivered in early May to alert the
PO to do the roll over again and also to aid in tracking the fault.
theis incident happened in mid may. What was the point of the
code delivered in 5_2?
Passing to EPOSS-FP to explain to John exactly what has been
delivered to CI3R in the way of diagnostic code for this issue.
F} Response :
This validation was release in WP 7865 on the 4th April 2000 from
development.
F} Response :
Development have given you an answer, but I'm not sure that it
helps.
What do you think?
F} Response :
1. I need to know what the correct Cash Account figures should
have been were it not for the Dataserver failure. Can these be
derived from the transactios in the message store, or the trial cash
account.
2. The diagnostic code which was delivered before this incident
happened was promised to aid in investigating the cause of this
problem. Has this code helped? How?
3. At some point a WP was delivered that would alert the user that
there was a problem with the SU roll over and the user woulf be
promted with a message to re do the roll over. Has this been
delivered? If so why did the roll over process not cease and promt
the user to try again?
I have been asked by Walter Wright to submit more detail and I
also note John's queries; in response to these first:
1. We can reconstruct the Cashaccount at some point but I do
not believe this to be an 'L1HOT' issue, I think this will have to
queued up and reprioritised (or cloned specifcally for this
issue).
2. The diagnostics have been useful for PINICLs such as this
becuase have have confirmed what we have suspected, in
that records have failed to be retrieved from Riposte calls
(when they work perfectly well in development).
158
Post Office Horizon IT Inquiry
EXPG0000001
EXPG0000001
Expert Witness Report of Charles Cipione, dated 14 September 2022
EDSC 2000-08-08
(Jim Anscomb) 14:07:20
EDSC 2000-08-08
(John Ballantyne) 14:35:34
EPOSS 2000-09-13
(Gerald Barnes) 14:18:44
EPOSS 2000-09-13
14:43:18
(Gerald Barnes)
3. Code has been issued at C14 which will back the user out from
key phases of rolover should the system detect that rposte
readied retrievals have failed to yield data.
I don't think I'm being premature in revealing that we think we
know know why these failures with Dataserver are occrring. Steve
Warwick experienced such a failure on a rig he was testing against
and found the root cause was that Archiving was active during a
riposte query; this only occurs ‘out-of-hours’ at the end of each
working day. Archiving will occur ‘in-hours' should the counter
have been switch off over night for 7 condecutive days and hence
the sprordic nature of these incidents (or where PM's do their
balancing near the archiving time at 10pm.)
F} Response :
PRESCAN: Diane's away, Steve Warwick is definitely not looking at
this call, need to check out what to be done as corrected CA
details may be required. Any problems contact L. Higman.
F} Response :
I have spoken to Martin McConnell who advised call to be routed
to EPOSS-FP for assistance to re-produce the Cash account as per
John Moran's requirements.
F} Response :
It proved to be very difficult to resurrect the cash account data for
week 5. Steve Warwick's analysis tool showed that not only was
stock unit DD corrupt but also stock unit XXX. EPOSS nodes
91579999 and 90029999 were missing and had to be resurrected.
In the end the reconciliation code was adapted to give data for
every CashAccLine with the exception of 99990001 which is the
receipts balance bought forward; but that can be calculated by
looking at the receipts total from the previous CAP CashAccLine
99990700. The resurrected figures are given in the attached file
CAPS. The lines containing
<Application:EPOSSWeeklyDump>
<DumpOf:AccumulatedFigures>
give the recalculated values for each CashAccLine. They contain
the CashAccLine number with a prefix giving the table number.
Note that lines 99990701 and 99990702 can not be trusted
absolutely but their sum will be correct for the overall discrepancy
table value. An alternative way of looking at the results is to look
at the lines
Containing
<Application:EPOSSWeeklyRecon>
<EPOSSTransaction: <TranType: WeeklyCAErr>.
They give the original and recalculated CashAccLine data for each
line that was wrong - all other lines in the cash account would
have been correct. Note that lines 99990701 and 99990702 are
not included in this set.
F} Response :
I am not sure it is worth spending time trying to resurect the other
CAPs. The method I have derived assumes that the CashAccLines
for the previous CAP. I see from Steve Warwicks's analysis that
CAP 6 was not correct as well. Now if I rerun the tool I have
developed on CAP 6 it will use as its base line the CashAccLine
figures in CAP 5 which we know are wrong and I have just
recalculated. I think therefore that enough time has been spent on
159
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
EDSC
(John Ballantyne)
MSU
(QJohn Moran)
EDSC
(John Ballantyne)
Customer Call
My observations
Was the immediate
issue fixed?
Was a defect/ root
cause identified?
Was this defect/
root cause correctly
recorded in the PP?
Is there evidence
that this defect/
root cause was
addressed?
Observations on the
management and
closure of the issue.
Observations on
defect / root cause
his problem and it is not cost effective to proceed further. However
in future where there is a problem with just one CAP we should be
able to resurect the figures more easily.
2000-09-13 F} Response :
15:38:06 John can we kill this one off?
2000-09-14 Thanks for all the effort. For the time being I have agreed that
11:14:08 reconstructed cash accounts will not be needed all the time, but
only by special request of POCL.
I have already issued the final BIM report. As such please close
this call, and hope for the best with the CI4 code which should
make this type of incident very rare.
2000-09-14 F} Response :
12:30:27 As per Johns comments closing call
2000-09-14 Date and time complete: 14/09/2000 13:34:45
12:49:37 Service Complete (Confirmation) Received
It appears that the CAPS account was reconstructed but that the CAP6 and CAP7
accounts were not.
A root cause was identified, namely that the archiving process was active during a
Riposte query.
The defect code “14: Development - Code” was recorded in the text which is consistent
with the text of the PP.
The reference to release Cl4 suggests that they believed a code fix would address this.
It appears that the investigation into the root cause of this PP was carried out with
proper diligence.
A root cause was identified and there was an expectation that release CI4 contained the
proper detection and remediation procedures to prevent this issue from recurring.
160
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Appendix A - Inventory of documents relied on
Document Type
PinICL / PEAK
PinICL/ PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL/ PEAK
PinICL/ PEAK
PinICL / PEAK
PinICL / PEAK
PinICL/ PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL/ PEAK
PinICL / PEAK
PinICL / PEAK
PinICL/ PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL/ PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL/ PEAK
PinICL/ PEAK
PinICL / PEAK
PinICL / PEAK
PinICL/ PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL/ PEAK
PinICL / PEAK
PinICL / PEAK
PinICL/ PEAK
PinICL/ PEAK
PinICL / PEAK
PinICL / PEAK
Document Title
Pc0026873
PC0026195
Pc0027139
Pc0028477
PC0030182
Pc0027321
Pco028528
Pc0028263
PC0031280
PC0031549
PC0031834
PC0031939
PC0031947
Pc0032182
PC0031996
PC0032173
PC0033082
PC0033250
Pc0030628
PC0031636
PC0033152
PC0033173
Pc0028847
PC0034961
PC0033339
PC0035599
PC0035901
PC0031395
PC0035708
PC0034505
Pc0036526
PC0038771
Pc0027324
PC0031884
PC0032855
Pco029148
PC0031907
Pc0041919
PC0041477
Pc0045090
PC0045309
PC0045580
PC0034036
Pco04ss46
Pc0048718
Pc0048716
URN / URL
FUJ00027003
FUJ00027211
FUJ00027630
FUJ00029091
FUJ00029755
FUJ00029789
FUJ00029832
FUJ00029840
FUJ00030450
FUJO0030674
FUJO0030930
FUJ00030982
FUJO0031101
FUJO0031124
FUJ00031206
FUJO0031675
FUJ00032062
FUJO0032246
FUJO0032275
FUJ00032293
FUJ00032423
FUJ00032563
FUJ00034029
FUJO0034224
FUJ00034278
FUJ00034604
FUJ00034731
FUJ00034968
FUJO0035068
FUJO0036136
FUJ00036863
FUJ00037419
FUJO0038157
FUJ00038613
FUJ00039260
FUJ00039293
FUJ00039673
FUJ00040054
FUJ00040565
FUJ00042700
FUJ00043195
FUJ00045452
FUJ00045829
FUJ00046317
FUJ00046403
FUJ00062520
161
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Document Type
PinICL/ PEAK
PinICL/ PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
PinICL / PEAK
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Monthly report
Document Title
Pc0050081
Pc0051382
Pc0051361
PC0052148
Pc0053061
Pc0053216
Pco04s061.
Pc0054604
Pc00s9762
c0031313
Pathway Monthly Report- December 1998
Pathway Monthly Report - August 1998
Pathway Monthly Report - February 1997
Pathway Monthly Report - March 1997
Pathway Monthly Report - June 1997
Pathway Monthly Report - August 1997
Pathway Monthly Report - December 1997
Pathway Monthly Report -January 1999
Pathway Monthly Report - February 1998
Pathway Monthly Report - March 1998
Pathway Monthly Report - April 1998
Pathway Monthly Report - May 1998
Pathway Monthly Report - June 1998
Pathway Monthly Report - July 1998
Pathway Monthly Report - September 1998
Pathway Monthly Report - October 1998
ICL Pathway Monthly Report - April 1999
ICL Pathway Monthly Report - May 1999
ICL Pathway Monthly Report - June 1999
ICL Pathway Monthly Report - July 1999
ICL Pathway Monthly Report - August 1999
ICL Pathway Monthly Report - September
1999
ICL Pathway Monthly Report - October
1999
ICL Pathway Monthly Report - November
1999
ICL Pathway Monthly Report - January 2000
ICL Pathway Monthly Report - February
2000
ICL Pathway Monthly Report - May 2000
ICL Pathway Monthly Report - June 2000
ICL Pathway Monthly Report - July 2000
ICL Pathway Monthly Report - August 2000
ICL Pathway Monthly Report - October
2000
ICL Pathway Monthly Report- November
2000
ICL Pathway Monthly Report - December
2000
URN / URL
FUJ00062974
FUJ00064777
FUJO0065150
FUJ00066141
FUJO0066464
FUJO0066611
FUJO0067416
FUJO0068089
FUJ00073008
FUJO0075020
FUJ00058198
FUJ00058158
FUJO0058160
FUJ000S8161
FUJ00058162
FUJ00058163
FUJO0058166
FUJO0058168
FUJO0058169
FUJO0058170
FUJO0058171
FUJO0058173
FUJ00058174
FUJ00058175
FUJO00S8176
FUJO00S8177
FUJO0058181
FUJO0058182
FUJ00058183
FUJ00058184
FUJO0058185
FUJO0058186
FUJO0058187
FUJO0058188
FUJO0058189
FUJO0058190
FUJO00S8191
FUJ00058192
FUJ00058193
FUJO0058194
FUJO0058195
FUJO0058196
FUJO0058197
162
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
EXPG0000001
EXPG0000001
Document Type Document Title URN / URL
Monthly report ICL Pathway Monthly Report - July 2000 FUJ00078051
Monthly report Monthly Joint Implementation Report - NFSP00000065
Sept 27, 1999 - Oct 24, 1999
Monthly report Monthly Incident Review - November 2000 POLO0029221
Monthly report Monthly Incident Review - March 2000 POL00029222
Background CS Support Services Operations Manual, FUJ00079816
materials version 3.0 dated 07 February 2000
Background ICL Pathway Customer Service Incident FUJ00079865
materials Management Process, version 1.0 dated 13
November 2000
Background End to End Support Process, Operational FUJO0079897
materials Level Agreement, version 2.0 dated 17 June
2003
Background NR2 HORIZON SYSTEM HELPDESK: FUJ00080410
materials Processes and Procedures Description,
version 1.0 dated 15 June 1999
Background Horizon System Helpdesk Call Enquiry FUJ00080486
materials Matrix, version 1.0 dated 13 March 1997
Background PinICL Incident Management Process, FUJ00098253
materials version 3.0 dated 30 January 1998
Background PinICL User Guide, version 0.1 dated 15 FUJ00098255
materials February 2000
Background PinICL Reference Data Guide, version 2.0 FUJO0098258
materials dated 18 February 2002
Background Horizon System User Guide / Balancing POLO0038868
materials with Horizon Guide (“HSUG"’}, version 1.0
dated 28 July 2000
Background Technical Environment Description (“TED”), FUJO0079645
materials version 4.8 dated 22 October 2002
Background Horizon OPS Reports and Receipts (“HRR”), FUJO0119554
materials version 8,0 dated 08 August 2000
Background HNG-X Architecture - Counter Business FUJ00118200
materials Application (“HXA”), version 5.0 dated 04
August 2017
Background Horizon Online Induction Training POL00089726
materials (‘HOIT"), which is not dated but is believed
to have been produced in around August
2009
Background IMPACT Release 3 Counter Design for FUJ00085124
materials Balancing, Rollover and Stock Processing,
version 2.0 dated 12 September 2005
Background Impact Release 3 - Balancing and Trading FUJO0085125
materials Statement Production User Interface,
version 2.0 dated 31 October 2005
Background IMPACT Release 3 Design Proposal, version FUJO0088336
materials 2.0 dated 20 December 2004
Background Explanation of Local P.O. Reconciliation and FUJO0079193
materials Administration
Background Operations Manual - Branch Trading: POL00086704
materials balancing and despatch’, version 7 dated
December 2006
Background Horizon Online: Introducing Horizon Online POLO0086712
materials
163
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
EXPG0000001
EXPG0000001
Document Type Document Title URN / URL
Background Branch Trading Transition Guide, dated POL00089708
materials September 2005
Background EPOSS Functional Description, version 4.0 FUJO0079277
materials dated 03 March 1999
Correspondence Submissions on behalf of Fujitsu Services FUJO0119556
Publicly available
document
Publicly available
document
Publicly available
document
Publicly available
document
Publicly available
document
Publicly available
document
Publicly available
document
Publicly available
document
Publicly available
document
Publicly available
document
Publicly available
document
Publicly available
document
Publicly available
document
Publicly available
document
Publicly available
document
Publicly available
document
Limited dated 13 September 2022 (in
response to a Rule 9 Request dated 29 April
2022)
The ‘Terms of Reference (updated)’ for the
Post Office Horizon IT Inquiry
The ‘Completed List of Issues’ for the Post
Office Horizon IT Inquiry
The seven phases of the Inquiry
Bates & Ors v Post Office Ltd ((No.3)
"Common Issues") [2019] EWHC 606 (8)
(15 March 2019) (Common Issues
Judgment)
Bates & Ors v the Post Office Ltd (No 6:
Horizon Issues) (Rev 1) [2019] EWHC 3408
(QB) (16 December 2019) (Horizon Issues
Judgment)
Technical Appendix to Judgment (No.6)
“Horizon Issues”
Appendix 2 - Summary of Bugs, Errors,
Defects - Judgment (No.6) “Horizon Issues”
Appendix 3 - Glossary - Judgment (No.6)
“Horizon Issues”
Post Office - Our Purpose
Fujitsu case study - Fujitsu's Systems and
Operational Services to UK Post Office and
the Worldwide Trend of Post Offices
Fujitsu case study - Post Office Limited, 07
November 2007
Post office numbers, 21 February 2022
Select Committee on Trade and Industry
Written Evidence - Appendix 16 -
Memorandum by Post Office Ltd, 12 May
2003
Securing the Post Office network in the
digital age, November 2010
Wikipedia article
Apple press release, 09 January 2007
https://www.postofficehorizoninquiry.org.uk/publicatio
ns/terms-reference
https://www.postofficehorizoninquiry.org.uk/publicatio
ns/completed-list-issues
https://www.postofficehorizoninquiry.org.uk/key-
documents
https://caselaw.nationalarchives.gov.uk/ewhc/qb/2019
/606
https://caselaw.nationalarchives.gov.uk/ewhc/qb/2019
/3408
https://www.judiciary.uk/wp-
content/uploads/2019/12/bates-v-post-office-
appendix-1.pdf
https://www.judiciary.uk/wp-
content/uploads/2019/12/bates-v-post-office-
appendix-2-1.pdf
https://www. judiciary.uk/wp-
content/uploads/2019/12/bates-v-post-office-
appendix-3-1.pdf
https://corporate.postoffice.co.uk/en/purpose-
strategy/purpose/our-purpose/
https://www. fujitsu.com/downloads/SVC/fs/casestudie
s/uk-postoffice2. pdf
https://www. fujitsu.com/uk/Images/postoffice-
customer-experience.pdf
https://researchbriefings.files.parliament.uk/document
s/SNO2585/SNO2585.pdf
https://publications.parliament.uk/pa/cm200203/cmsel
ect/emtrdind/718/718we17.htm
https://assets.publishing.service.gov.uk/government/u
ploads/system/uploads/attachment_data/file/31809/1
0-1260-securing-the-post-office-network.pdf
https://en.wikipedia.org/wiki/List_of_best-
selling_mobile_phonesi#1999
https://www.apple.com/uk/newsroom/2007/01/09App
le-Reinvents-the-Phone-with-iPhone
164
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Document Type
Publicly available
document
Publicly available
document
Document Title
Review of the Internet Watch Foundation -
A report for the DTI and Home Office by
KPMG and Denton Hall
Ofcom Report - The Communications
Market 2004, 11 August 2004
EXPG0000001
EXPG0000001
URN / URL
https://webarchive.nationalarchives.gov.uk/ukgwa/199
91013043222/http://www.dti.gov.uk:80/iwfreview/iwfr
eview1.html
https://web.archive.org/web/20090905171759/http://
www.ofcom.org.uk/research/cm/empdf/cmr04_print/c
m_2004. pdf
165
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Appendix B - Example Attribute Grammar
Shown below is an example of an Attribute Grammar:
<Message: <Groupld:23422><Id:8> <Num:363><Date:22-Jul-
2000><Time:10:37:11><User:SGRO01><Expiry:35><TxnData:<Container:FF>><EPOSSTransaction: <CutOff
1D:20><TranType:C><Summary: <SU:FF><CAP:18><BP:02><SV:1250.6><Qty:4><LTSV:1>><PM:<L1:><
L2:><L3:3181><L4:><L5:>><SM:>><CRC:EB8BFO00>>
<Message: <Groupld:23422><Id:8><Num:480><Date:22-Jul-
2000><Time:11:05:42><User:SGRO01><Expiry:35><TranStartNum:480><TxnData: <SessionId:23422-8-
480><TxnId:23422-8-480><Container:FF><Start: <Date:22-Jul-
2000><Time:11:05:26><TF:5>><End: <Date:22-Jul-
2000><Time:11:05:39><TF:1>><Mode:SC>><Application: EPOSSAppMain > <EPOSSTransaction: <ProductNo:
262><Qty:1><PVer:33><SaleValue:377.3><BlackBoxData: <M:SC><UnitPrice:377.3><S:1>><AdditionalDat
a:<ACC_NO12:[REDACTED]>><TranType:S><PM:<L1:><L2:20><L3:3006><L4:3013><L5:3017>><SM:><
Discounted: False> ><Credit:37730><CRC:4D7F7DA6>>
<Message: <Groupld:23422><Id:8><Num:531><Date:22-Jul-
2000><Time:11:34:09><User:SGRO01><Expiry:35><TranStartNum:531><TxnData: <SessionId:23422-8-
531><TxnId:23422-8-531><Container:FF><Start: <Date:22-Jul-
2000><Time:11:29:37><TF:9>><End: <Date:22-Jul-
2000><Time:11:29:46><TF:4>><Mode:SC>><Application: EPOSSAppMain> <EPOSSTransaction: <ProductNo:
261><Qty:1><PVer:23><SaleValue:46.4><BlackBoxData: <M:SC><UnitPrice:46.4><S:1>><AdditionalData:
<ACC_NO12:[REDACTED]>><TranType:S><PM:<L1:><L2:20><L3:3006><L4:3013><L5:3017>><SM:><Di
scounted:False>><Credit:4640><CRC:AC9978C7>>
166
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Appendix Ci - Example PinICL
Shown below is an example of a PinICL with some of the challenges of interpreting this highlighted as call-outs.
We have deliberately selected an example PinICL which contains more fulsome descriptions (some PinICLs are a
lot more challenging to interpret). It is important to note that a PinICL was written for internal tracking purposes
by the team trying to investigate and resolve the identified issues. They were, presumably, not written is such a
way as to provide a complete and accurate explanation of all investigatory steps such that someone reviewing
these some 20+ years later could fully understand what had occurred. The PinICL should be viewed with this
context in mind, but nonetheless the challenge of interpreting these documents remains relevant to my review
and therefore the approach adopted.
Figure 18.6 Example PinICL (1 of 8)
Reference to other PinICLs
Expor Pc00445: Typos, shorthand and
= emmary = acronyms that have to be
— deciphered
Gemma
—ny
Current release (version) of
I_}-—*+ the system
Call transferred
multiple times within
a ticket
167
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 18.7 Example PinICL (2 of 8)
Reference to work performed with
no detail of what this entailed
17055 8 Deaton, Referenced attachment is
Ca account not available for PinICLs
Call transferred
eS multiple times within
a ticket
I
\U
yn 8.0128 Caerne Che
ne 80127 Cae Chg
ees 150128 aee hee
ne 0128 Care Cheng
32080128 Cavern chee
esr 26 ete ean
Further acronyms
I_I that need to be
understood.
\
Apparent internal
disagreement as to whether a
declared discrepancy could
give rise to a receipt and
Paget payments imbalance
Figure 18.8 Example PinICL (3 of 8)
Reference to attachment that are not available in
the produced data as this is a PiniCL.
“ senmany one Atsptte tomer te
teas se roan att
roms conraoucoriorra I ovesnmssator amie ais» meMemyraeistins Wor & Deen
=aaakaw on lene gia
cr 0 mento tmoornotance sma ;
0 1600) ren evento sce in ter en Further acronyms, in
‘epee hn br fg opty tr wate I this case a reference
I to a Stock Unit.
Reference to issues /
situations with no
additional detail (e.g., “the
1* Class stamps
revaluation)
e700 183728 Cater hey
seyenew 332728 Camere te
Call transferred
1704200008129 Lonel gran See multiple times within
pono 82235 ent ngnan a ticket
September 1 Paget
168
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 18.9 Example PinICL (4 of 8)
Reference to another
attachment that was not
produced and therefore not
available
x
£., = — fs se hacia Detail of transactions and
products without view of
=< noe ‘ee a the audit log files being
- referenced
Call transferred
multiple times within
a ticket
mee OSS grt nin corer mo mp
mor revmenct 2728
Reference to other PinICLs
I
Figure 18.10 Example PinICL (5 of 8)
wt sennery omee anceps ater prose re
text sone vents
asus corr ig yt evenn ate eam 158 witennnink OS etn Call transferred
TSN lore SSS SS ST multiple times within
fnew 40853 Lontngnn teeta anata I] a ticket
pronynetarnrna pephpmunrelrenneersioneener
rem) wate wd ocareen ster coc te OR
rem einie want osresneneronrate ne eer ny
sieves) en teeta eat nga 730 oh a te Raforancerto
ewe investigation into
comttd I_I message store, which is
I not available for our
same ren review
yoyren 15470) Sete Yom arc atu ace geet Rang at rout eet
2am 070 Sane cf at wh eh re arid
September 2 perry
169
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 18.11 Example PinICL (6 of 8)
Indication that this is a known software
issue that will be fixed in a future
release (C14)
Protect crue
See Mosca
roots coy rzuor a nt trom ceen
jp —eenmnnintasmeiale as I} Reference to other PinICLs
sevemesure Sense ontcntyeeg br thine A I
en Renee
onde eee Cte bt aed owner
Further investigation shows
the previously detailed root
cause and resolution is not
accurate
Oo
Call transferred multiple
times within a ticket
I
I
Figure 18.12 Example PinICL (7 of 8)
Reference to other PinICLs
um
unt
copy roan? tao Neots
cee o ayet
peed Last eptte atone
(eso east 4/7/200017219 Me Many 72639632774
Probe Gro
S77
sparen 11823
moe
ow
aerate
wer wate
oer ete
—— =
eorenencr ssn
mente on Cary $0 eset ne mesg
Y
Call transferred
multiple times within
a ticket
eee I
Detail of similar issues
sisepemee 3
Paget
from other PinICLs
Reference to other PinICLs
Further investigation
provides another different
root cause and resolution
170
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 18.13 Example PinICL (8 of 8)
Reference to other PinICLs and
cloned calls
mt senmary pened Last optote cmomer Protec roe
toaent Oy sons rose At Fatt
00172928 es One
(7700145339 cao Ore
(70 145239 Catan Ore
Se ee me
ae
Call transferred
multiple times
within a ticket
Minimal detail as to
resolution of the issue,
instead using codes and
ronan : categories
(2700172138 oneagran 2 Du ct Cato 6 Te
\_/
September 2021 rence
171
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Appendix C2 - Example PEAK
Figure 18.14 Example PEAK (1 of 2)
Acronyms often used that we
had to lookup (e.g., EDSC)
Peak Incident Management System
Call Reference P('0045077 Call Logger _Customer Call_-- EDSC
Release Talgeted At - CSR-CI3_2R Top Ref E-0005 162468
Call Type ine Incidents Priority B -- Business restricted
Contact Call Status Closed -- Reconciliation - resolved
‘Target Date 1970572000 Effort (Man Days) 0
Summary Raising call on the back of closed call e-00050908
Progress Narrative
Pate:t6-may-2000 16;
fran #0004507 opened
700 Uaex!_Custoner Call
re708/00 33
1 was closed in erfor a0 raising ehis call with closed call upaates in ic P___I_ Ticket refers to another
closed call
format
his is a syeten incident for 5:
frepert (national totais) shows 33tpe delays enter cate
(75/00
farvea: date 4/3/00. please investigate and rescive.
by cazrett simpaon at i2-may-2000 15:44:00 category 40 -
Tnvastigation There can be up to 500,000 APE
Ticket is advised to close then
I gets reassigned but no
detailed explanation
squeated a0 call
bissncatician rane:
frustoner opened date 26/05/2000 27:29:52,
Patai7-yay-2000 09;aa100 Ueeriarbara Longley
updated te CR-cT3R
de
outing call back to Garrett Simpecn in EDSC #0 that call can now be
srogressed correctly.
iexo OF REFERENCE 10155184)
Fesponded to call type L ap category 40 ~incident Under investigation
the reaponse was delivered to: Powerelp
Repetitive / copied & pasted
text throughout the ticket
pe Las Cateaery te nrnertene Uadee eaves eset eer
dial Caspase) uss GalivetedI tor Dosernaip
sre :28 vay 2000 09:30100 Georscarsote Sinpoon
‘Row that fuscher derail absut these missing cesngactions shouls be
watiebie fron your 10-s0y sepo
atiii missing, please tell me which FAD ip
172
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Figure 18.15 Example PEAK (2 of 2)
Ticket gets transferred to other teams /
team members multiple times within a ticket
concerned and 1 can Snvestigate further,
f the beans 1s have now turned up chan please close
ews oF nerenence 12432943)
pended to call type i aa category 40 ~incident under inveatigation
can.
javei 25-May-2000 09:33:00 UsariGarrett Simpson
che response was dalivared co: Ponertels
te Sale casera tas been teanafeczed tothe Teant MU-rads tat
jours epent since call received: Ob
saverid-gun-2000 107
jours spent since call received: 0
TOO WaarrAngels Shaw
igned to the Team Member: Angele shen
heures
feusry 05/07/00 17:06 gpvezise WSu1 Information: 12/06/2000 21:37:56 - By
wgeie shaw
the GALL Secnrd has boon
gels Shaw
sates fom pinch.
igned bo the eam Meabers
Rs
mall assigned to Angela Shaw - MSU-Indt Mgt
eno oF nerenence 20173695)
egondied tn eal Ege
he response wee delivered
TT=Sad-2000 14158700 Uesriiarbara Longley
pon
Cateyery 40 —Taniilent Wider Tevantipetin
eo: PowerRelp
T-mag 2000 OFF
she Salt sescrs haz been
rargee Relat
fF) Response
the Gali zecord has been assigned to EDEC Zean Menber: Garrett sinpscn
feo oF Rerenence 21230525]
sesponded to call type i as Category 40 -Incident Under Investigation
she Zesponge wes delivered to: PonerHelp
Fi Rag F000 O8TDTI00 Uae harbarm Longley
updated to CSR-CT3_2m
arerT-mag-BO00 TIs48100 UneriGarrete Sinpa
} Reapons
array 42 requested clon
eno oF REFERENCE 21247058)
jessie BS) eave oT
ISurs spent since call ceceives: 0 hours
Cae 50 ieee elteat eer
"aver @i-Aug-Z000 11149100 TesrrGarrett Simpson
“ALL PC0045077 closed: Category 30, Type L
he respons ne aelsvares oo!
Root Cause General - Unknown
Logger _Customer Call_ -- EDSC
Subject Product APS -- (version unspecified)
Assignee Deleted User -- EDSC
173
EXPG0000001
EXPG0000001
Post Office Horizon IT Inquiry
Expert Witness Report of Charles Cipione, dated 14 September 2022
Appendix C3 - Example KEL
Figure 18.16 Example KEL
HORIZON KEL rcoleman3549n
KEL type: Information
Title: Have message exceeded CM_10_WAIT
‘Summary: Have message exceeded CM_10_WAIT
Raised: by Richard Coleman on 11/10/1999
Last updated: by Richard Coleman on 08/01/2004
Release: 13
‘System product: Counter
Keywords: index, disc, disk, CM_10_WAIT, An unrecoverable er
Status: ‘Authorised
Visibility: Medism
Peak: C3108
Tis: 9910090003
Version: 1
Symptoms
Riposte error DEMODIFY operation falled. The Vo completion wait operation timed out (exceeded CM_10 WAIT} <be><br>An
unrecoverable error occured wrhin the cache manager. The /o completion walt operation timed out (exceeded CM_10_WAIT)
(0xC10A0020) The message server will be shutdown abnormally. <br><r>An error occurred while waiting for an Vo completion
‘on volume 1 for unit LPN 35168. The /o completion walt operation timed out (exceeded CM_IO_WAIT) (OxC10A0020
Problem
Probable disc problems
Solution - Helpdesk
Reboot counter<broif message reappears then send engineer. <broif 2 Single Counter Outlet engineer Is to replace the mirror
disc first and If the message reappears to then replace the base unlt.<br><br><kel><b>BEFORE</b> the base unit 's replaced
{ot 2 SCO) the anger <B>MUST<[te contact the SMC to eynchronice the wessagestores troGee KE PCeNDIS2 for
further details. </kel><br><br>On 2 Mut! Counter Outlet replace the base untt.
Evidence
None. Raise no calls.
174
EXPG0000001
EXPG0000001